That’s the one. I am afraid you’re going to have to get into the formulas in the “methods” section at the end if you want to really understand the lateral connectivity in the model. That’s where I’m at.
If you read the columns paper assuming that the L2/3 is doing hex-grids fits very nicely in the descriptive text and in some ways, suggests hex-grid behavior - for example: while L2/3 and L5 cells exhibit “complex” RFs (Hubel and Wiesel, 1962; Gilbert, 1977). Key properties of complex cells include RFs influenced by a wider area of sensory input and increased temporal stability (Movshon et al., 1978).
Cells which have similar classic receptive fields when presented with isolated edge-like features, diverge, and fire uniquely when the feature is part of a larger object.
To explain border ownership, researchers have proposed a layer of cells that perform “grouping” of inputs. The grouping cells are stable over time (Craft et al., 2007).
- I am certain that the function of L4 works as a timing coordinator with the interface with the thalamus for the formation of waves and local synchronization to these waves. I have supporting papers on this but that that is not the issue I am working now.
I predict that the timing below will be related to the gamma rate (40 Hz) or 25 ms:
Activations in the output layer do not require very fast inhibition. Instead, a broad inhibition within the layer is needed to maintain the sparsity of activation patterns. Experiment evidence for both fast and broad inhibition have been reported in the literature (Helmstaedter et al., 2009; Meyer et al., 2011).
Our simulations do not model inhibitory neurons as individual cells. The functions of inhibitory neurons are encoded in the activation rules of the model. A more detailed mapping to specific inhibitory neuron types is an area for future research.
Wait - what was that last bit? …
The functions of inhibitory neurons are encoded in the activation rules of the model. A more detailed mapping to specific inhibitory neuron types is an area for future research.
The biology seems to point to smaller local pools of inhibition that are triggered locally. This is not rocket science - a little stroll through related papers should lay out the scope of inter-neurons receptive fields and modulating output connections.
While I am banging on non-biological problems with this paper here is a monster one:
In this experiment, each column receives lateral input from every other column.
This is absolutely NOT how it works in biology.
These connections are the foundation of the Hex-grid formation and the model skips right over it.
What does the biology say?
From this paper in my hex-grid post:
What Proportion of Layer 3 Pyramidal Cells Receive Long-distance, Excitatory, Monosynaptic Inputs?
Our findings also suggest that most pyramidal neurons in layer 3 are targets of long-distance, horizontal projections. Specifically, low-intensity stimulation at long distances from the recorded layer 3 pyramidal cell evoked monosynaptic EPSCs in the majority (77%) of these neurons. However, this proportion is likely to be an underestimate since some long-distance axon collaterals were probably severed by slicing of the tissue blocks. Although the present study does not indicate whether horizontal projections synapse selectively onto layer 3 pyramidal neurons, our results show that these cells frequently receive this type of synaptic input
I have several more paper (including ones referenced by Numenta) that support the exclusively longer length of lateral connections. When you consider this go back to my third drawing showing a “halo” around a given mini-column in the post above.
And then consider how that drives the formation of triangular formations.
So the answer to hex-grids questions was “we considered that - go read the columns paper” and “sure it would work but why would we look at that?”
- How about - because the biology does it?
More comments on the columns paper:
Hex-Grids are strongly compatible with most of these with no changes.
Ok, I read the methods section again - it is what I saw the first time I worked through the math.
Here’s where they play footloose and fancy-free with the biology - the tangle of axons that fan out from the mini-column is what activates the inhibition cells - not some vague inhibition field in the model.
Look very hard at this related bit in Computing Cell States
As I said above: this section totally ignores the known topology of interaction between the mutual reciprocal connections and the range and activation properties of inhibition inter-neurons. When it comes to interactions between neurons I can’t stress this enough: Topology Matters. As offered in this paper the model continues the earlier simple “thousands of synapses” models and ignores the actual biology.
As described, the calculations based on this model is correct. The problem is that the model is NOT based on the known topology of the biology so it is misleading - it does NOT serve to explain the behavior of the biology. Numenta claims to be biologically inspired to explain how the brain works; they will have to do better.
The hex-grid proposal faithfully models this aspect of the biology.
I would start with modifying the model so that lateral connections are all at a biologically plausible distance from mini-column to mini-column. Your suggestion about looking at the hex-patterns that form should produce some very interesting behaviors.
I agree that Numenta has identified the inhibition function for future consideration. Why this is important is the bit I have mentioned before - the tuning of the ratio of distances of the mutual connections and inhibition range modifies the behavior of the column. At one ratio the column acts a Gabor filter - exactly what is needed for early sensory processing. Reduce the inhibition range and the column acts the hub of a hex-grid formation. I would think that this range of behaviors would be very important to the studies of cortical column theory.
He had not considered the idea until I presented it to him yesterday. Once he understood it, he said it was not necessary for sensor fusion.
He understood how hex grids might emerge across minicolumn activations via CAN / mexican hat. He did not see the utility for our current work.
It pains me that you are disappointed, but this is really about Numenta working on this problem from a completely different vector for the past 10 years than you’ve been. We have been in the weeds, inside a cortical column, inside a layer, inside a minicolumn, figuring out detailed interneuron mechanisms to answer some core neuroscience questions and get a foothold into this common algorithm. That’s where we started and that’s where we still are. The ideas you bring to the table are certainly worthy of thought, discussion, scrutiny, and investigation, but they just don’t cross the direction we are heading.
I want to point out that both Jeff and Marcus saw the mechanism and understood it (much faster than I did). And neither denied that it was interesting and could be playing a role in computation at some level. But it is simply not overlapping with the questions we are currently asking about object representation.
Please go back to your grid video and watch it with your new-found understanding of hex-grid coding.
Really, I know you made it but, watch it with your new eyes.
Then come back and tell me that there is no overlap with what Numenta is doing.
In particular, how else with HTM make phase/rotation/scaling the way hex-grids does it.
Grid cell modules in entorhinal cortex and these hex grids are similar, but not the same. They both express the same characteristics, but their mechanisms are very different. Grid cells in entorhinal cortex do not exhibit the physical topology (hex grid bumps) that you describe. They form grids, but the grids are not physical hexagonal shapes formed by the placement of cells in the cortical substrate. Their bumps only exist in the space being represented.
We will use the grid cell tricks we can already see happening.
I will keep slogging along on the track I am on, and Numenta, on it’s track.
I assume that they will cross over at some point.
I will keep up with the forum and help the nubies as I have been doing.
I am still onboard with HTM as described with the BAMI paper.
Were we diverge (Numenta and me) is the understanding of the distribution of object representation.
At this point I see Numenta looking with a laser focus on the single column and trying to explain as much of known behavior as possible with a local mechanism.
I have the different insight that this is a cooperative process between the various maps and in streams of connections. I see that this allows a much simpler local function and also see many connections to papers that I have read about this distribution of function.
Some of my foundation beliefs that drive this viewpoint:
- The contribution of the subcortical structures in selecting behavior. (My dumb boss/smart advisor model)
- The three streams paper is the closest to what I think happens in training the hierarchy.
- I also really like Randall O’Reilly’s take on the prediction model in L5; it compliments HTM.
- That the web of map interconnections is critical to understanding individual map functions.
- The basic Global Workspace mechanism in connecting the need state to the perceived state.
- My model of consciousness based on cortical connections, primarily the Arcuate fasciculus
- The basic model of the hippocampus as a one-day buffer for personal experience
- The contribution of subcortical structures in adding affective behavior guided by the amygdala.
- That this affectively seasoned personal experience is transferred to the cortex in sleep
- Most likely using spike timing learning.
- That the basic HTM model is the best description of the local cortical computation and state transition.
- That “thinking” is sequences of motor commands directed to maps.
- That the cerebellum is important for guiding sequential actions inside the brain. (in addition to the usual motor actions driving the body)
- That human speech is a learned motor action (generation AND perception)
- That the learned motor programs of speech form many of the functions attributed to “higher mental actions.”
I see these as interlocking and forming a fairly complete model of human cognition.
When these are considered as a whole the local function of the cortical column is fairly well constrained
I thank you. I’ve been working more closely with the research team lately, so I will be sure to keep an eye out for places where this particular bag of tricks could be applied. And if it does ever emerge from the research team, I’ll be sure to point out where it truly originated.
Bitking, I think you should write something for neuroscience researchers. You’ve been accumulating knowledge for decades. For the sake of understanding how the brain works, neuroscience is sorely lacking summarization. I shouldn’t have to spend a thousand hours just to understand the details of a single cortical layer, and I imagine the situation is similar up there at the level of maps and such. Heck, not just the details. Learning some essential things takes so so much sifting through papers. It’s endless rabbit holes.
So whether or not your ideas end up working and being accepted, I think you have a lot to offer to neuroscience and indirectly, in the short or long term, AI. Even just outlining what sources exist and what they are on in a peer reviewed article would be very helpful. Summarize decades of reading and the contribution would seem to me as great as the findings by famous researchers. You’d save hundreds or thousands of years of reading.
Planning to write a review article is how I deal with my lack of contribution to HTM theory, anyway.
Agreed. In the very first few lines of my hex-grids post I made that clear:
First - let’s be clear what I am and am not talking about.
This is about cells that work by forming regular hexagonal grid structures and are not necessarily the same thing as the “grid cells” that are coding some spatial aspect of the surrounding environment.
These grid-forming cells that I will describe combine a sparse collection of mini-columns into larger hexagonal arrays. The cells that form a regular hexagonal array can offer a separate and highly useful tool in the neural network toolbox. They can act to take a local response from a mini-column responding to some locally sensed condition and join it to other local mini-columns that are also responding to what they see of their part of some locally sensed pattern. This is a form of lateral binding.
The Moser/grid cells use these hex-array forming cells to code the external sensed environment so we have a hex-grid signaling an external location (in a grid pattern) to the place cells in the hippocampus.
What in your opinion would be a good path to test out the hex-grid forming layer approach?
At least in a sense of seeing whether it fits within HTM, can be built on top of it as you mentioned in the original hex grid post. I’m certainly wiling to give it a go programmatically, but whether it turns out to be the correct approach is another thing.
At any rate Im glad you pointed out this particular path of research, especially Calvins work Bitking!
The lowest hanging fruit is to take the code referenced in the columns paper and change the lateral connections.
The goal is to see groups of htm modules stepping through whatever learned patterns in parallel both in learning and replay.
Each htm mini-column would be learning different things but through the lateral connections these would be considered parts of a larger pattern. (Binding)
You don’t need to change anything about the current HTM lateral connections, you can just run a process on top of it the activated minicolumns within a region.
You’ll have to ensure that there is a 2D topology enforced across all cortical columns that includes minicolumns. Each minicolumn must be able to efficiently calculate direct links to neighbors. And the calculation for that link needs to be programmable so you can tweak parameters for the CAN application.
For every active minicolumn, search for neighbors that are also active. At the end of the search you should have many grids of varying size. The larger the grid, the stronger the link with the SDR input.
This process could work within a cortical column, or across a population of cortical columns.
I understand the proposed technique.
The all-to-all connections misses the interlocking sparsification and competition that arises from the hex-grid pattern proposal. This feature allows competing perceptions to attempt to match a larger spatial pattern; this is one of the important features of lateral binding of patterns.
Are you saying this hex grid application is contained within local groupings like cortical columns?
Another area where a hex-grid forming layer could be applied is to improve the implementation for the Output Layer referenced in the columns paper (this is the area I have recently been exploring). The paper describes selecting a random SDR to represent each new object during a training phase which is required prior to performing object recognition. Hex grid formation could in theory be used to form these SDRs online from activity in the Input Layers, to eliminate the manual initial training phase. I believe this approach, if done correctly, should also capture object semantics as well (where similar objects have proportionally similar overlap in the representations in the Output Layer).
Consider this example of the possible set of input patterns on L1 that might be found on a given section of cortex:
We would like the trained mini-column to vote on what pattern they might be sensing and settle on one of these sets of output patterns on the projecting axons from L2/3:
So in this given section of the cortex (spanning hundreds of mini-columns) the alignment between these input and output pattern is this:
I want this input pattern on L1 — > to form this output pattern in L2/3:
This samples a tiny subset of these mini-columns in this area. The actual number of input-output pairs co-existing in this section of cortex could be a fantastically large number. Naturally, the number of activated grid elements is related for a given pattern to the size of the learned pattern; as time goes on the pattern could get larger as more “fence sitting” elements are induced to learn the edge cases.
Note the important binding feature that allows mini-columns with very limited dendrite spans to work with very distant mini-columns to learn a very large continuous pattern. Sparsity is preserved at both the dendrite and the mini-column levels.
Are you arguing that these hex grids span across regions? Like from V1 to S1?
No - Calvin tiles work across large spans in a single map. In this case, I am focusing on the top of the hierarchy in a single sensory modality.
The H of HTM will work acoss maps. I have some very concrete proposals for that but this part of the theory (Calvin tiles) has to be elaborated before showing how the map-2-map parts work.
Yes - there is more.
Spot on @Paul_Lamb! Exactly what I was thinking for some time. Glad I am not the only one.
My perspective is that hex grids seems like a more bio plausible alternative to temporal pooling that also has the potential to solve the binding problem / sensory fusion. It may also be a way to unify / merge the activation on distant cortical columns without explicit synapses between them. I think there is a lot to discover in this last sentence. I imagine I can only grasp a fraction of what this implies.
On the other hand, my main concern is the reduced capacity that results from hex grids because of the imposed local constraints.
I hope hex grids will have its say in my experiments soon.