HTM Hackers' Hangout - Jul 6, 2018

Even in On Intelligence, Jeff thought the classic view of hierarchy was correct, but after learning about grid cells and realizing how they might be used in the neocortex to model objects in allocentric spaces, we had to rethink the hierarchy to make it work. HTM theory posits that a location signal (via grid cells) exists for each cortical column representing some unique point in space. This assumption broke our classic idea of hierarchy if we continue to apply the Mountecastle model of a cortical column.

If we assume every cortical column is representing some feature at some location in an object’s space, this just doesn’t make sense in the classical view of hierarchy, because each region of the classical hierarchy is identifying different objects and composing them. We don’t think it works like that anymore. This new model of hierarchy explains a lot of things, including object composition by association. stay tuned for podcast

If there was a black box in front of you and I asked you to reach in with your hand and tell me what it was quickly, it would take you about 0.5 seconds to grab the coffee cup, recognize across your hand hierarchy at all levels that it matches coffee cup, and tell me right away.

However, if I asked you to reach in with your foot and tell me what object was inside, it would take you much longer, and you would probably find yourself touching and thinking “what was that feature, a rim? oh, that’s a cup then”. That’s because your foot hierarchy has never learned coffee cups. It knows only the world of feet, like socks and shoes, lint, sand, coins, and the occasional unfortunate wad of gum. That part of your brain has no models of coffee cups unless you teach it (like every other part of your brain).

So how did you recognize it with effort? Think about the hand and foot hierarchies as different structures that eventually converge together at the high levels. Your foot has no idea what a coffee cup feels like, so it passes simple features like rims and curved planes and smoothness up the hierarchy until it can match it across columns that link downward in their hierarchies to different sensory areas!

That is the 2nd essential point here, that cortical columns train each other as they learn. That’s how we think this learning transfer happens at all levels of the hierarchy. Just like this, at lower levels of the hand hierarchy your fingers train each other as they learn things. That is how you can touch something with one finger to learn it and recognize it with another finger, even on another hand. This lateral learning must be happening across cortical columns at all layers of the hierarchy.

To be clear, we don’t know exactly how this works yet. But we have thought about it a lot and are working on solving it.

1 Like

Spatial pooling, I think, is happening within each column where sensory input is processed, but I am not sure it is happening to the feedforward input from lower regions.

1 Like

True, it might not be necessary if representations in the output layer are sufficiently sparse. I was just not making any assumptions about the pooling algorithm used (in HTM research code, for example, there is a “Union Pooler” algorithm which uses 10% sparsity IIRC versus the typical 2%). Also keep in mind that the number of minicolumns in the input layer is likely less than the number of cells in the lower output layer (depending on the configuration of course), so that could be another reason to use a SP.

1 Like

So to summarize what I am getting from the various answers:

  1. Columns and hierarchy start out as is usually proposed in most takes on brain theory.
  2. As learning progresses the internal model is proposed to be refined and pushed out toward the sensors. This has the possibility to reduce processing time and reduce the processing load in the association areas.
  3. Some of the process to push the learning down may include sampling relative position. This part is not backed by any proposed mechanism but a firmly held intuition that this could explain many observations.
  4. The position grid mechanism communicated by hex-grid signalling has been identified as a possible lead to working out this unknown mechanism as it has been observed to signal spatial position in the EC.
  5. This same hex-grid signaling has been observed in hubs of the various lobes, putting it closer to the sensory input streams, therefore the spatial signaling may also be available at these points.
  6. This position signal available at the lobe hubs may be pushed out via some non-hex-grid signaling format that is yet unidentified.
    Q: Why this last point?
    A: Because full-on hex-grid signaling has not been reported from the primary sensing areas. It may be there but I have not seen it documented anywhere; I have been looking for for this for a long time.

Am I getting this right?

1 Like

4:39

Why is it important to change the size of the sensor array relative to the column size?

Why not change the size of the sensory array and leave columns at the same size?

I don’t want to put words in other peoples mouths but I believe that this is an attempt to move beyond the +/- 8 column reach of the dendrites of a single cell and allow binding of the features of an object over a larger area.

Binding is a central problem in neural network representations. How do we combine all these individual features sensed by individual columns into a larger object?

This is one of the main driving forces that lead me to Calvin’s hex-grid coding method as outlined in my “HTM to hex_grid” model. This model allows a unified object representation to be bound over a large area of a map using all local operations without resorting to the magic of the inter-map connections to do representation spreading. If tract bundles diverged as is sometimes proposed that would be a plausible method but study of real brains shows that they maintain loose topography as the tracts connect one map to the next.

What @Bitking said above, but also this matches what we see in biology. Higher regions in the hierarchy are still getting feedforward sensory input. Why? The classic hierarchy model had no good explanation of this. In this model it makes sense that larger columns at higher levels of the hierarchy will have larger receptive fields within the entire sensory array, which gives it the ability to see “the whole picture” while also sampling the details of the picture to confirm the feedforward input from lower regions are in sync with the object under scrutiny.

And it gives us a way to continue to apply Mountecastle’s big idea, which was that every column is performing essentially the same task. This model of hierarchy in some ways generalizes out the idea of scale from the problem of object representation. I’m not sure if I’m expressing that correctly.

1 Like

I had you until #4 above. Maybe we are not thinking of grid cells in the same fashion here. At least we never use terms like “hex-grid signaling” at Numenta. We say “grid codes”, which are usually unique SDR-like representations of space (originating from grid cell modules), which can be used as a starting point for object definition giving continuing sensory movement within the object space.

1 Like

I invite you to compare the hex-grid post to observed EC behavior without preconceptions and see if you see the tight correspondence that I do.

I see the information signaled and the coding method as separable.

I’m not sure we have as many preconceptions as you. To accept this hierarchy model, you only need to assume two things:

  1. grid-cell mechanisms can produce virtually limitless unique representations of space as SDRs
  2. transitions between these representations can also be represented in SDR

If you assume this, you pick a unique location in “the universe in your brain” to start defining an object. Then, through movement and sensory perception, build the model of the object in space using only movement vectors, or transitions, from one point to another.

Every cortical column produces unique representations of space. When you choose a random location to start defining a new object, you are doing that in tens or hundreds (thousands?) of cortical columns. Each one chooses a different unique location to start defining the object. They are all different. The only thing that is compared is the end result, the object representation, which is detached from the sensory input/location.

To make this idea work, we needed do scrap the old hierarchy model. When we started rethinking it, this made much more sense and opened up a lot more doors for us.

2 Likes

You are responding to the data being coded. We are in agreement here.

I am chasing after the coding method to represent this information. As I just said - I see these two things as separable.

Yes, I agree with that. But I also think that problem does not involve the hierarchy. I don’t think learning “concepts of space” necessarily requires hierarchy to occur. I haven’t talked to anyone about this, but why would you need hierarchy for grid cell behavior to emerge?

1 Like

I don’t have a dog in the “grid cell behavior” fight.

I have been reading much literature on this as I run into it with google searches and checking the references in these papers and to the best of my knowledge - nobody really knows how the underlying spatial codes are generated. Could be local or it could be hierarchy but as I said - nobody knows yet.

There are many theories at this time and nobody has demonstrated unconditionally that the proposed model has verification in animal studies. I have not committed on this yet.

I do have to point out that there is a vast body of research that documents the connection patterns of the various maps and there does seem to be regular pathways if not hierarchies. Likewise - I have not seen any research that refutes the pioneering work of H&W as to the simple and complex cells as you move from V1 onwards. Likewise in the auditory pathway

Many studies have shown that when a spot of activity is generated at some area in the EC it tends to be a patch of hexagonal connected cells. These are thought to be attractor basins but even that is not fully confirmed.

A few studies have remarked on observing this same hexagonal coding in the various hubs of lobes so I am making the bold leap that the same coding method is at work in both places.

1 Like

I am staring to really like these grid cells. The logic of it is that they really work
very well with spiking daisy chain loop logic. Very flexible. I have made shift counters,
memory shift shift counters. memory units. Up and down counters. program pointers with
jump forwards or backwards to the next loop. Kind of like a all purpose FPGA.

2 Likes

I think every region does the same thing, but anything close to the scale of the grid cells in EC takes hierarchy.

Lower regions seem to operate on such sort timescales that it’s hard to see them doing much processing beyond a single contact. Moving the fingertip to somewhere else would be enough for primary regions to forget the object.
So I think primary regions apply the same mechanisms used for object recognition to recognize features. Imagine the system’s fingertip darts around poking different features. Primary regions can still apply the same mechanisms to a single feature. A single poke does not produce a single sensory input because it takes a little while on neural timescales, so it’s like there are multiple features on the primary cortex’s version of an object, even if that object is as simple as a corner.

1 Like

I just got a lot of feedback from Jeff about this video, and the next version is going to be a lot different. This exercise has been really useful, though. And your feedback is crucial for making these visual aids valuable.

1 Like

In the animation I see what seem like the component columns growing in size as they go up the hierarchy. I thought it was only the hypercolumns that increased in size the higher up the hierarchy and not the component columns that compose the hypercolumn. That is I thought higher up the hierarchy hypercolumns were composed of even more component columns and that’s why they were bigger. Is there a source to indicate that component columns also grow in size?

Where did you get the terms “hypercolumn” and “component columns”, those are not terms we usually use.

Well, there was minicolumn and hypercolumn used in some of the literature. From what I hear minicolumns do not necessarily correspond with a specific function, but there is something that is often called a column that forms part of the larger structure, I think some of these were called orientation columns for example. For example in the orientation tuning, the idea was that there were structures tuned to particular orientations and while there was continuous change between these, there was a minimum chunk containing a set of all possible combinations.

http://hubel.med.harvard.edu/book/b28.htm#ori

A cortical column, also called hypercolumn, macrocolumn,[1] functional column[2] or sometimes cortical module,[3] is a group of neurons in the cortex of the brain that can be successively penetrated by a probe inserted perpendicularly to the cortical surface, and which have nearly identical receptive fields.[citation needed] Neurons within a minicolumn (microcolumn) encode similar features, whereas a hypercolumn “denotes a unit containing a full set of values for any given set of receptive field parameters”-Cortical column - Wikipedia

The picture is obviously more complex than what the earlier literature proposed. But I’ve assumed that in recent literature showing differences between column sizes, it is referring to the conceptual hypercolumn. Since I’ve heard that the columns composing a hypercolumn do not necessarily correspond with minicolumns, I’ve chosen to call them component columns.

EDIT:

Particularly the numbers in the following quote suggests it is the hypercolumn that is being said to vary in size.

On examination, they found that rat vibrissal cortex comprises about 500,000 neurons, the number and 3D distribution being remarkably preserved across animals. More importantly, the number of neurons per cortical column varied between 10,000 and 30,000 within individual animals.Neuroscientists find cortical columns in brain not uniform, challenging large-scale simulation models « the Kurzweil Library + collections

Thanks for the clarification. We were just talking about the size of cortical columns: