Universal Encoder

Science Numenta Theory
1

Hi Everyone,

I feel I need to clarify that the idea that I proposed was to create encoders that will act as the primary input channels analogous to a human beings senses. The idea is have sensory encoders that require no pre-processing and can take the world’s state in directly as pieces of a “Universal Encoder”.

I’m hopeful that it makes sense that we can feed in auditory; visual; tactile (kinesthetic); optical character recognition; ascii character streams (for reading the internet) - etc. directly because that is what human beings do. We could convert these “basic” formats to SDR encodings and never have to write another encoder.

All the HTM does (just as all the neocortex does), is process sequences of patterns. From this raw input we build a vocabulary and conceptual model of the world. I’m hoping this makes sense because it seems we could use the series of Hierarchies which turn the world’s sensor data into SDRs which represent higher level concepts - and re-use them such that when an HTM user wants to build a complex Hierarchy to act as an “intelligence” to satisfy a specific purpose; that person could start out using a section of hierarchy (or hierarchies in the case of more than one input “sense”), that processes the format of input necessary to accomplish that specific task?

A Universal Encoder - is what @karchie was speaking of in the original thread? (I can’t help hearing Universal Translator from Star Trek when using that phrase :stuck_out_tongue: )

4 Likes
2

How is a universal encoder biologically plausible? Our brains don’t have one. The senses we primarily depend upon (sight / vision) have very complex encoders going on in the retina and cochlea. :confused:

3

Sure our bodies have one. I’m talking about sensory input. We don’t have a team of Data Scientists massaging and normalizing our visual information before it come into our eyes? In that same way, an HTM could just merely take in the raw data - provided we create sensors that can take it in and transform it to SDRs which represent that data. I’m taking about the digital equivalent of our senses as a replacement for Encoders.

Once we have these basic senses or Encoders, then we don’t have to create specialized ones.

I just re-read your post…

The senses we primarily depend upon (sight / vision) have very complex encoders going on in the retina and cochlea.

These “complex” encoders are what we need to create. …and we don’t have to limit ourselves to the human repertoire of senses either - we could have more - such as a Geospatial sensor or something…

4

@cogmission we have encoders for every sense. The retina is a dramatically evolved encoder, which produces around 30 specific channels of transformed information by using very complex circuitry. The auditory system includes physical feedback-controlled hair cells which actively alter the signal before it even gets to the sensory hair cells, and then there are about 12 stages of processing which we only vaguely understand after 60 years of research. Even pain and touch signals are actually coming in from the brainstem where transformation occurs.

The idea is easy when you think about the fact that reptiles and birds don’t need a neocortex, and many of them can be said to have what we would call some kind of “mental life”. The encoding has already evolved in animals without a cortex, and what we mammals mostly do is just plug in to wherever is most appropriate in an existing, fully functional brain system.

So, maybe there is a “universal encoder”. It’s called a reptile.

5 Likes
5

I think the eventual goal is to remove the necessity for developer intervention when creating HTMs for a particular task. It would be “nice” to eventually be able to only be concerned with “designing hierarchies” rather than creating encoders and massaging parameters… It will come eventually, but its kind of nice to “dream” about where we’re headed?

1 Like
6

I thought keeping a collection of shared encoders would be the way to do this. Honestly at this point in NuPIC’s existence, I was hoping there would be more community-created encoder experiments. (If there are an I don’t know about them, please post them!)

3 Likes
7

Here Here! Let’s create more for sure!

What I’m talking about Matt is way down the road… Keeping a collection of shared encoders is the initial stage of what I was talking about, and absolutely necessary (I agree) at this stage in HTM development. :wink: I’m just dreaming about 20 years from now…

8

This article I just googled is interesting on the bird neocortex analogue subject @fergalbyrne . Interestingly also talking about analogues in some species of turtle:

Exactly what kind of mental structure reptiles experience is an interesting question though. I wonder if anyone has done experiments. Googled it. Yes. Inconclusive but interesting:

Maybe, like the turtle in the cortex analogue article above, reptiles do rely on similar structures to get similar functionality, they just don’t have as much of them as the neocortex.

In response to @karchie 's original question about the importance of pre-processing. I thought the examples given in that Andrew Ng talk I posted the other day of the visual cortex receiving input from tongues(!) and auditory “click” echos practiced by some blind people, was particularly striking as evidence for the flexibility of the cortex, and its probable centrality for the important parts of “higher level” functioning:

My original point with that post was to argue that movement was necessary to “linearize” input to the visual cortex, and push it into the time domain. But the MRI evidence of visual cortex involvement with this echo location is interesting on its own terms:

“When the participants couldn’t move their heads or torsos, they zigzagged down the virtual hall and were unable to self-correct before hitting a “wall.” When the corridor’s position was fixed and their bodies and heads were free to move, however, the novice echolocators soon righted themselves”

Also @karchie I’m arguing all through this that causality (as presented by Jordan some weeks back) or generalization over prediction/context are universal encoders.

-Rob

1 Like
9

This appears to be a good paper on this topic:

Cell-type homologies and the origins of the neocortex

“Our findings establish that the layer 4 input and the layer 5 output cell types are conserved across the amniotes, but are organized into very different architectures, forming nuclei in birds, cortical areas in reptiles, and cortical layers in mammals.”

Proc Natl Acad Sci U S A. 2012 Oct 16; 109(42): 16974–16979.
Published online 2012 Oct 1. doi: 10.1073/pnas.1204773109
PMCID: PMC3479531
Evolution

1 Like
10

Matt christened this branch thread “Universal Encoder”. I’m talking to the theme that the important encoding is done in neocortical like structures across species. (Actually still relevant to @karchie 's original “The art of SDRs” thread arguing SDR’s need to be heavily preprocessed, but that’s labels for ya.)

Anyway, here’s another amazing paper on common neocortical structures across cognitive species. Murray Shanahan at UCL has broken down the connectivity of an entire avian neocortex analogue:

Large-scale network organization in the avian forebrain: a connectivity matrix and theoretical analysis, Murray Shanahan,1,* Verner P. Bingman,2 Toru Shimizu,3 Martin Wild,4 and Onur Güntürkün5, Front Comput Neurosci. 2013; 7: 89.

Among other interesting aspects they encoded the entire connectome in a connection matrix and ran some network stats on it. Since I’ve been looking at @mraptor 's matrix abstraction of the CLA (http://ifni.co/bbHTM.html) I wonder if it might not be interesting to run some of the same stats over the connection matricies he derives. That would provide a concrete comparison of HTM networks to biology.

Apart from providing a potential test for our own network connectivity, and providing a neocortex analogue home for encoding outside mammals, the contrastive biology thing is excellent for drawing attention to what may be truly important in the neocortex. It turns out in birds functional groupings of cells are not localized in the physical sense (maybe not so easy to interpret MRI’s a la Tom Mitchell in birds?), and not layered. Suggesting layers in the neocortex may be only artifacts of cell types and not spatially functional. Instead what is conserved contrastively are “small-world networks” (most nodes not neighbours, but only a small number of nodes apart.)

I think “small-world networks” are what you would expect if connections code sequence. Not all states can follow each other, so you would not expect universal connectivity if connections code sequence.

So, perhaps this contrastive evidence from the corresponding structures in birds supports the idea synapses code sequence (and cells associated with similar functions connect richly, also “small-world” = temporal encoding??), but that physical layers in the neocortex may not be functionally significant (suits me, because I want to find hierarchy in the predictive connectivity, a perfect fit for observed “small-world” network structure.)

2 Likes
11

Sounds interesting. Could be a topic for HTM Hackers :wink:

12

Ha ha. Could be. You’ve picked up that I don’t like labels Matt. My disbelief in labels has a deep relation to this whole subject of cognitive processing :slight_smile:

@mraptor 's implementation looks like a software issue, but it provides a different perspective on the theory too. I was struck to see a matrix extraction connectivity network come with a deep relation to theory in Shanahan et al.

For HTM there could actually be three of four papers in that: HTM matches observed avian telencephalon connectivity stats, etc (if it does!) Just need to run mraptors bbHTM over a few data sets and then perform the network analysis Shanahan et al detail in their paper on his connection matrix.

13

Rob, that is certainly an interesting paper, but the connection analysis it presents is at the region level (between 52 regions), not at anything like a micro-column level.

Also, any connection graph can be expressed as a matrix; whether it is stored that way in practice is just an implementation detail.

14

@robf Thanks for the echolocation link. I’ll guess that the superior colliculus is playing an important role in this; it’s a subcortical nucleus that in humans is involved in saccade and gaze shift planning and maybe attention, but in bats it’s important for echolocation. SC gets inputs from lots of modalities and feeds to the LGN (among lots of other places), and, interestingly, visual and auditory input are space-aligned within the SC. So I’ll claim it’s not surprising that visual cortex is active in blind people doing echolocation: visual cortex gets (indirect) input from SC (which in blind people is mostly location information from audition), and not much input from anywhere else; input-deprived cortical areas pick up whatever input and function they can (cortex being apparently a general-purpose computing substrate, hence our interest in HTM); so a task that’s really working the SC will drive activity in visual cortex.

I don’t think this says anything about sensory encoding for HTM, because the early auditory system is doing the heavy lifting to produce clean, cortex-friendly input, and the midbrain is doing sophisticated integration of auditory and somatosensory input to generate essentially exactly the right representation for echolocation.

Side note while I’m thinking about audition: location information from auditory input is interesting in that cortical neurons could not extract this from the raw signal because they’re too slow: there needs to be specialized hardware to handle the sub millisecond coincidence detection (a fast K+ channel and specialized dendritic structure, at least). Which is not an argument that HTM couldn’t in principle learn a universal encoder; just that the biology and physics of this particular case require a specialized solution in real brains.

15

@floybix Sure Felix, the way information is presented can be seen as an implementation detail. The same way a choice of programming language can be seen as an implementation detail. But it can also be very clarifying.

For instance, I think HTM has been too stuck in the physical geography of the brain. Particularly layers. It’s obscured what is important.

Look at your own implementation. Tied up with the incidental complexity of regions and layers. If you liberated yourself from that implicit object model and thought more about the implementation detail of the connection matrix, the underlying simplicity of the problem might be more apparent.

There is a simple vs. easy argument here. (Notice I’m riffing Rich Hickey here, I’m not really blaming you Felix, you’ve just copied the implicit object model which has also held back HTM, but the irony is just too amusing for me. There’s another layer to it too, in features, but enough for now…) It is easy to naively base your implementation on the physical arrangements of the neocortex. The avian telencephalon is nice because it challenges the importance of that contrastively, and draws our attention to where the true simplicity of cognition might lie.

Let’s look at the problem as if the essential simplicity is a connection matrix. It’s even clarifying things for me. I have a bunch of perspectives in terms of the connection matrix which I’m going to try to present here. But enough for this message.

1 Like
17

Wow, that’s interesting that the superior colliculus is involved in saccade and gaze shifting in humans, but in bats it’s important for echolocation. Thanks. More evidence for common vital role in spatial perception. I think they are both trying to push the signal into the time domain which is the universal encoder (prediction generalization.)

Anyway, that apart, you’re dismissing the fact that the visual cortex can work with varying sources of information as evidence that sensory coding is not of primary importance.

Could be. Given dozens of different input sources, you could still argue they have all, separately, evolved to do the same job.

But why would they all separately evolve to interface with this other thing which came later? After all , the other thing evolved very differently in other respects (e.g. spatially distributed rather than localized.) Why wouldn’t it have adapted itself (especially as it evidences localized specialization in mammals and functional specialization in all) to multiple independent meaning representations, evolved separately by the different sense organs, rather than requiring them separately to conform to it? Even if there is one universal set of meaning features which they all separately evolved to, why wouldn’t they all code them differently? Why wouldn’t one just randomly turn out to have chosen big-endian, and another little-endian?

I’m sure you can still find an argument why they would. But why bother? We have a candidate universal encoder (for a resolution of the world into meaning “objects” anyway) in the cortex (prediction generalization.) We don’t have one for all these other sensory inputs. Why not use the one we have rather than hypothesize many that we don’t have?

P.S. On your argument that cortical neurons are too slow to capture some information captured by the hearing system. I don’t doubt the sense organs add information. I just don’t think they are the primary source of meaningful interpretation.

BTW, I’m going to develop this later. But since you’ve been vocal just these few days raising the lack of hierarchy in HTM (quite rightly, we need a bit less spin and few more tough questions in HTM), let me just point out a hierarchy is implicit in the connection matrix. I’ve been arguing this for a while: that hierarchy is implicit in the connections (contexts) and we should forget superficial physical layers. Formulation in terms of a connection matrix makes this clearer. We can argue about how relevant this hierarchy is. But suffice to say for now, it is there, and it might be the one we want. It is something nice about this connection matrix perspective that it exposes us to this. The connection matrix might be a good way to access the ideas. So, yeah, the cortex temporal network gives us a candidate universal meaning encoder (prediction generalization) and it’s one which has an implicit hierarchy for you too.

18

@robf you have it the wrong way round. The cortex evolved to tap in to existing encodings in the reptilian midbrain, which already generates combined sparse encodings of sensory input. That’s why we have to build encoders for HTM systems - the cortex is not capable of handling raw sensory data directly.

1 Like
19

@fergalbyrne not to doubt you, but do you have any sources that confirm your last statement?

20

@franky yes, sensory input (apart from olfactory) to the cortex comes through the thalamus, where it is combined with signals from Basal ganglia, other mid- and fore-brain signals, and feedback from cortex. Most signals from the Cranial Nerves and Ascending Sensory Pathways pass through additional nuclei before reaching the thalamus. In particular, the integration of retinal and other subcortical signals carried out by Superior colliculus project strongly to the lateral geniculate nucleus in thalamus which projects to visual cortex.

1 Like
21

Thanks @fergalbyrne I was under the impression there were dual connections, one through thalamus and one direct, not sure where I got that from.