Universal Encoder

Science Numenta Theory
65

I hope I’m contributing to the discussion by drawing attention to just what it is in the brain that needs to be encoded. But I may be off topic.

@abshej wrote “images as processed from the ocular input (only these qualify as vision)” as if that implies ocular input is in some way unique in being information in a visual ‘format’ or tagged with a marker unique to ocular input.

While @cogmission wrote “neocortical processing centers use the same algorithm”. A view supported by the report of a blind person’s non-visual input being processed in their visual cortex. Cogmission goes on to suggest “downstream processing… takes the signals and uses them to produce olfactory, visual, taste, kinesthetic and other sense information”

Taking all three points together implies that input is first ‘tagged’ and then processed with a universal algorithm before being rendered downstream into output in the sensory modality specified by the ‘tag’.

I’m a newbie. I assume such ‘tagging’ in the brain is replicated in HTM by the encoder’s properties of sub-sampling, classification, and union.
Could someone tell me:
1] Is there any evidence of this ‘tagging’ or ‘formating’ occurring in the brain?
2] And is it the same sort of process by which the information which forms allocentric models is ‘tagged’ with location?

66

“I’m a newbie. I assume such ‘tagging’ in the brain is replicated in HTM by the encoder’s properties of sub-sampling, classification, and union.
Could someone tell me:
1] Is there any evidence of this ‘tagging’ or ‘formating’ occurring in the brain?
2] And is it the same sort of process by which the information which forms allocentric models is ‘tagged’ with location?”

Bob,

I will go out on a limb and state that this “tagging and coding” is given by the fixed processing path. All the processing paths are formed locally by the cell to cell connections with dendrites & synapses, and for long-range connections, by bundles of fibers.

A thing to keep in mind - bundled connections from one area of the brain to another are fixed during development; there is no hardware switching of processing paths. There are dynamic patterns of activity that use these paths that are part of perceiving and responding. See page 57 in this link to see how the dynamic patterns of activity use these fixed paths make the connections you are asking about.

The connections locally at the ends of these bundles of fibers after the animal is formed is determined by the growth or shrinking of synapses - what we call learning. This shapes how the patterns of activity will be formed and routed.

The information patterns follow fixed paths & topographical targets set by the physical development that is determined by the genetic coding in the homeotic section of the genome.

The morphogenesis process gets progressively more detailed as the development unfolds.
http://brainvis.wustl.edu/resources/VE_Nature_97.pdf

There is a veritable smorgasbord of individual chemical markers that drive this elaboration; some examples …


Just for fun - the process continues well after birth.
http://onlinelibrary.wiley.com/doi/10.1002/cne.902330309/abstract

There are a lot of details to keep straight but I hope this helps put the various parts in some sort of meaningful arrangement.

1 Like
67

That is indeed what I am saying. That ocular input is unique from other sensory inputs in the way it is generated and thus the way information is encoded in it. As far as the term format goes, I wouldn’t use it since everything is a bit representation(sparse or moderately dense).
I am pretty sure every sensory region is different from the other in some specialised way(still computing the same algorithm). These differences must be due to fact that the data it processes has different features than other data, encoded differently. So that specialised region can process other types of sensory data too but not as efficiently as it will after some time, during which I suspect the architecture pertaining to the connections will change considerably.

I think this is correct. Apart from the connection pathways there is no tagging in the real sense of the word that happens to the data. But despite this the differences in the data cannot be ignored.

68

I think this can happen if the person could see after birth and lost his/her sight later in life, after which he/she was using the tongue-sensation module. Since, then the person can associate some visual inferences with those coming from the tongue module. Plus, the idea that the sensation coming from the module are given to the cortical area processing visual stream might make this possible. But, again, this means that the cortical area associated with and adapted to process tongue data cannot do visual inference despite our way to force the tongue to encode that kind of data.
If the person was blind since birth, I find it very hard to believe that he/she could “see” in any way.

I concur. :hushed: Though I find it difficult to get myself to tag Mr.Hawkins and Mr.Ahmad in posts. :upside_down_face:

1 Like
69

I don’t think the neocortex differentiates between the senses that strongly. You learn your model of the world using all available senses. That model still exists if you suddenly loose a sense. As you continue to observe the world through your other senses, those models you built up with missing senses will still exist and continue to hold meaning.

2 Likes
70

Agreed. Despite this, the model is being created using specific sensory perceptions. Those perceptions are not all same, despite the almost common processing. The differences do matter. What I was talking about is the details of those perceptions. For example, hearing certainly isn’t the same as seeing. You can infer that someone has entered the room using either sight or sound, but sight is different from sound. The way you experience or perceive both is drastically different. One encodes different type of data and also lot more data than the other.

Yes, but the absence of the missing sense does make a difference in the perception.
And also, if the agent doesn’t have the particular sense at all since the beginning then the model of the world is very different from the one that an agent with that particular sense will have made.

71

I recently re-read a year old post from my blog that talked about the principles behind encoders.

Let’s explore how we might produce an SDR of a particular pattern in our naive version of the sensorimotor engine:

Let the first sensory representation Maestro receives be CAT. Each one of those letters represents an aspect of the spatial pattern: CAT. If we were to convert this to an SDR we’d start with the C and make a binary index for that. Meaning if C is ever found in the first index you would have the SDR start with 1. So the representation CAT would be 111. Not that sparse, but just wait until we start to see more things. If PAT is the second representation Maestro sees we add on an index for the P that we’ve never seen before: 0111 (not C, then A, then T, P). The mapping is not contained within the SDR, that would be some kind of encoder’s job that is doing the work of conversion. Eventually, as we see more and more things our representation becomes more and more sparse, and looks more like 0000010010000000010000000, ideally approaching about a 2% sparsity.

This is an adaptive SDR and it may not be immediately clear why an adaptive SDR is valuable. Well, maybe it’s not, but here’s where I’m coming from:

Regions of our neocortex use SDRs (as instantiated by neurons firing). But the process by which they are used requires that the data be pre-formatted as semantically encoded and that it can fit in the representation space. This is handled by the sensory organs in the body. Your skin is touched, or your eyes see an image and the way in which the sensory neurons in your skin are activated produce a semantically encoded representation of the sensation.

Of course, these processes by which information is semantically encoded by the sensory organs before the impulses reach the neocortex is a gift of evolution. That’s the algorithm by which our sensory organs have been so finely tuned that they deliver the best possible semantic representation of the data to the brain. The information about how these things should be semantically encoded has reached from the brain (where it is held) back through the algorithm of evolution or selective reproduction to influence how the information is gathered to better suit the organism’s brain. It all goes together. But maybe we could do this without evolutions help, maybe we could produce an auto-encoder explicitly. I think we’d need an adaptive SDR in order to achieve this.

What I think I was trying to say is that the sensory organ is an extension of the brain in so far as the brain can influence it through evolution, which takes time.

I see any kind of “auto-encoder” or universal encoder as an iterative function that must learn alongside the HTM structure and be influenced by it.

Step one might be making an adaptive SDR that grows when it sees different representations. So this is an expansive force upon the SDR’s length.

Step two might be some kind of algorithm that notices which bits always shine together, they can be consolidated into one bit. This is a contractive force upon the SDR’s length.

Step three might be to discover simply through an auto-encoding neural net, or by some other means which bits tend to be near each other in space and time. that way we can arrange shallowly-similar representations to have overlapping bits in the SDR.

Step four I think would require more in-depth analysis derived from the HTM model we’re feeding. once it learns some higher order similarities in the data, (that is, that representation A which shares no overlapping bits with representation B are actually semantically close), the HTM model needs to be able to export that information back down to the encoder explicitly, (rather than waiting for the algorithm of evolution) to run it’s course.

if it could explicitly communicate that idea back to the encoder, and know which way the encoder would modify it’s bit representations, and therefore be able to modify it’s own internal HTM connections, then we’ve bypassed evolution, or rather we’ve imported the effect of evolution into the intelligence algorithm directly. It’s still iterative and takes time to train the encoder, but it’s adaptive over time. We’ve pulled in the encoder to be the lowest layer of the HTM algorithm at that point.

Anyway, I have no idea if this could work but my intuition says there needs to be a feedback mechanism between the HTM and the encoder because there is a feedback mechanism in nature called evolution, which produces the effect of a “universal” encoder, because the encoder, the organ, becomes attached to the brain through the process of evolution and is, therefore, an iterative learner with it.

My intuition says just connect the two explicitly instead of through the round about way of using a genetic algorithm to produce the best winners over time.

2 Likes
72

Exactly.

1 Like
73

How will the HTM system itself know that one encoder is better than the other? Basic HTM in and on itself will work on any SDRs given to it. The only way I see to get better encoders using the HTM structure itself is using genetic algorithms.
Also the encoding has to be done on and using the raw data and not the SDRs generated. The features of the data cannot be guessed by the network and it has to process whatever SDRs it receives in order to extract the features. I don’t see how it in itself can influence the encoding process of those features for better encoding.

1 Like
74

I think genetic algorithms might be the best place to start. We should start with recreating what actually works right now. But I imagine that if the HTM can deduce that some bit representation from the encoder is semantically similar to another bit representation from the encoder that the encoder represents without sharing any bits, well, the HTM can send that data back down to the encoder and modify it. Therefore it may converge on at least a local minimum of semantically encoding overtime.

75

While this may be so, and I am not sure about it, there is no way for the running HTM to know whether the semantics it is verifying correspond with the actual world, since we might start with any random encoding process. Since HTM will work on any SDR generated by a random encoder, there is no direct way for the HTM to test whether the encoded ‘features’ in the SDR actually match with the semantics of the real world object. Especially when the sensory modalities the HTM might compare the inferences between, using voting, differ very drastically. This is because SDRs share spatially similar features because of the way they are encoded. So the SDRs about the same object generated by two encoders that encode very different information about the world(or same information differently) need not share spatially similar bits. SDRs will share semantically similar bits only if they come from the same encoder.

1 Like
76

Ah my young padawan - you have much to learn!

in other places in the forum, we have been discussing the wonders of grid encoding. If all paths lead to transforming whatever stream into a distributed grid representation then there is a common format allowing SDRs to learn the relationship between these widely dissimilar sensory streams.

For maximum utility - the end results are interspersed in the same representational map; what would commonly be called an association area.

77

@Bitking Isn’t grid encoding exclusively related to location encoding?
And even if they map out which sensory pattern belongs to what in the map there is no way to verify their authenticity. Right?
PS. When can I be a Jedi?

78

Is there any reason that a method used in one sensory encoding would not be employed in another?

In the rat entorhinal cortex, we know that at a minimum it encodes vision, whisker barrel cells, self-motion, head direction, and vestibular sensations. All have been found to result in forming a location map.

Expand your idea of mapping - you map your way through your experiences. Some examples - navigating through a social interaction. Navigating through a book or story. A common memory trick is to place things in locations in real space - known as a memory palace. Some reflection on this can show that much of what you memorize has a spatial component. Relations (joining verbs) between things forms basic tuple to the relationship between nouns - a spatial relationship.

79

Yes, because they encode different information, even if it is about the same world.

Mapping is fine but a common map just unifies the patterns externally to locations.

Despite this, I don’t see how it gives a way to the method described in previous replies.
I am not denying the existence of spatial components in all patterns, that’s how they are encoded.

80

You ask about a universal encoder. I believe that is the central thrust of this thread.

Encode to … what?

So a universal encoder transforms whatever stream that comes to it into some common format that is readable by an SDR neuron.

The SDR is the reader of this code. What is it that an SDR reads? Conjunctions of spots of activity over a fairly short range.

If it is not intermingled grid patterns it is going to end up being something very much like it. A thought for you. If we know with a very high degree of certainty that a chunk of cortex works a certain way in one spot is it a stretch to expect it to work the same way in other places?

If it helps - the nervous system evolved to run bodies; to sense the surrounding space and relationships in that space and navigate through it. All external senses feed this basic mission. We have added some nifty tricks to this basic function but that has been the guiding principle from the start.

In the brain, we have a WHAT and WHERE stream. Each parses the raw information using different rules. The WHAT stream can parse semantic meaning by the grouping of like items. They don’t have to be physically adjacent - they only have to group in spaces that an SDR equivalent of a place cell can see them. See the cortical IO “retina” for a wonderful applied example of this idea in action.

In the WHERE stream, we already know the hippocampus reads the WHERE stream to learn places.

81

I agree with almost everything except a few details and the gist about the similarity of encodings.

A universal encoder will encode all semantic information from any type of data into an SDR(or a bit representation in general)

It might as well, but unless all encoders encode using grid cells that store semantic information, I dont see how all types of encodings are similar. And even if sensory modalities encoded using grid cells, I don’t think they will encode all semantics, it’s just not in the grid cell’s domain to do so. What I think is being discussed is that there is a reason why there are special cells and special, complicated processing paths for every sense organ and each sense organ encodes different kind of information. If there aren’t these differences then you cannot encode different meanings.

Encoders work separately and only create SDRs(or moderately dense representations). Those SDRs(bit representations) can undeniably be combined, split, used, abused, etc. by the HTM but that doesn’t change how they were encoded by the encoder in the first place.

82

Really - when it comes down to it - can you be more universal than either a single conjunction of what and where or the transition between a pair of what-wheres? Where does it say that encoding has to be a strictly parallel process? Part of this universal encoding is the sequencing of these things as the data streams in. A sequence or what-wheres is a unit of data. A sequence is also a description of a noun-verb-noun relationship of anything that can be perceived or imagined - an action if you will. The parsed input is transformed until you reach this relatively high level of description. The “letters” being seeming meaningless parsed patterns of bits until they map to the “words and phrases” of the WHAT and WHERE streams memory. The temporal part of HTM driving the transition of states in this stream.

The implementation being various hardwired encoders that are genetically tailored to whatever it takes to transform each stream. Each is transformed into some data format that is common to all encodings so that they can be combined in useful ways. A what and where stream. The joy of it all is that the system is designed to learn whatever pattern that it might encounter. It’s universal. I note that grid encoding is, in fact, capable of performing these functions and has been observed doing so in the wild.

If this is in fact how the brain does work I could see more of the WHAT stream fed to the side being the speech half, with more of the WHERE stream being parsed out on the other; the one doing spatial navigation and spatial sequences in general.

Combine this with the consciousness loop I described earlier - you would have yourself something interesting.

83

Maybe not. But this doesn’t pertain to encoders(hypothetically).

These various hardwired encoders are in question yes I think they have to be different for each data type.

This format is a bit representation or an SDR but again falling back to the main discussion, these are formed differently using the hardwired encoders that are genetically tailored. I argue that these encoders cannot be universal for all data types.

Will read that post.

You can still guess the gist of a sentence even if the order of the words are jumbled up, in most cases and you could also make a system with a single symbol representing a single ‘word’ and get rid of letters. The point being, this is subjective. The temporal nature is taken into account but the location is subjective. Correct me on this, I stand by whatever I said about universal encoders, but regarding this, I really don’t have sufficient information. Everything can be mapped with location but does everything need to be? The way we understand language could be explained without the grid cell logic, still using HTM, right? Our brain might have evolved to map everything the way you described and so everything gets processed that way, but it might not be functionally relevant in everything.

Maybe not a stretch but it is certainly not a given and without sufficient evidence with respect to other areas of cortex, there is no need or reason to assume so.

84

The cortex is not activated by a grid pattern but forms the grid pattern.
Look here to see how the columns do this feat with all the elements of SDR neurons.
It’s pretty much a one-to-one mapping. Please look at slide 45: note each column’s dendrites picking up some learned local part of a pattern and the winning columns inhibiting neighbors. This is all HTM canon law.


I wish I could say that this is all original with me but, truth be told, William Calvin has been teaching the basic elements of this for years. I don’t agree with all of his macro systems but at the micro level, he made a lot of sense.
COMPRESSING THE CEREBRAL CODE - William H. Calvin
http://williamcalvin.com/socns94.html
http://williamcalvin.com/bk9/bk9ch6.htm
The entire book:
http://williamcalvin.com/bk9/

1 Like