I’ve been digging further into some of the technical documents and whitepapers to get a firmer understanding of the current HTM theory. I am curious about a phrase that is used frequently, “layer X projects to layer Y”. For reference, I drew up a depiction of my current understanding of relationships between layers and regions
This is just based on the depiction that is used in several power point presentations, combined with information from the HTM Whitepaper (please point out any errors, BTW)
Anyway, one example is “layer 4 projects to layer 3”
Another way Jeff worded this in one of his presentations is,
If the system can predict what is going to occur next, we want to form a stable representation in the next layer, and if it can’t, then it passes those changes through. “Look, I’m not able to model this change … I can’t handle this, the next guy’s going to get these changes”.
I’m curious what the process for this might entail. In particular, I am curious what steps would need to be emulated by an implementation of the relationship between these two layers. In the HTM Whitepaper, it mentions that one possible implementation of layer 4 is one cell per column (so that it can only remember simple sequences) How would the original input and this output from layer 4 translate to the input for layer 3? Is there a different process for when layer 4 can predict what is going to occur next versus when it cannot?
Hi Paul,
There is a long history in neuroscience regarding the layers of cells in the cortex. It includes the types of cells in each layer, the connections between the layers, and the connections between regions. The central and widely held belief is that this connectivity architecture is nearly identical for every cortical region regardless of what the region does. A theory of neocortex should explain the function of each layer of cells and why they are connected together the way they are.
The diagrams I sometimes use (that you show) are much simpler than the real anatomy. It is common in the neuroscience literature to show more or less of the detail depending on the context. These diagrams aren’t wrong but leave out many details that are more subtle and often speculative. For example we are currently working at a much more granular level of detail that includes L2, L3a, L3b, L4, L5a, L5b, L6a, and L6b. These have all been distinguished in the literature and we are working at understanding the function of each. Our theory has progressed a lot recently where we now find we need these distinctions.
If you read journal papers about these connections you will find that “what connects to what”, heavily depends on how the connections were determined. It takes a lot of practice to interpret these papers. A good place to start if you want to delve into this is Alex Thompson’s 2007 “Functional maps of neocortical local circuity”.
To say Layer 4 projects to Layer 3 generally means all or most of the excitatory neurons in L4 send an axon into Layer 3 and form synapses with neurons in Layer 3. The axons typically split and go elsewhere as well, so it isn’t an exclusive connection In HTM theory it makes a big difference where on the neuron the synapse is formed. Generally, neurons in the cortex have three integration zones, proximal, basal, and apical, and the affect on the post-synaptic neuron varies depending on the zone. HTM sequence memory is the only theory I know of that explains the role of the different integration zones in the context of a large network of neurons. Unfortunately, most of the connectivity data about the cortex does not include this critical piece of data about which zones are the termination of axons.
This spring we had a major leap in understanding what region does and one consequence was I abandoned some ideas that I held for a long time. I still believe a region builds a predictive model and a stable representation, but the changes are not passed up the hierarchy. It is too much for me to explain here, if you are interested check out some of the material we posted earlier this year on sensory-motor inference.
Our current belief is the following. Layer 4 works nearly identically to the Layer 3 model we use in the HTM temporal memory. It has mini-columns. The big difference is that distal dendrites receive most of their input from L6a, not other L4 cells. L4 forms a unique SDR for different locations on an object whereas L3 forms a unique SDR for each element in a sequence. In our current model L3 and L4 share minicolumns, together they try to predict the next input either by high order sequences or by knowing the location on object determined by movement of the sensory.
Before the end of the year I hope to be able to give a much richer explanation of what is going on in the different layers of cells and what mechanisms are being used in the different layers. We are still sorting through the details. Hopefully this comment will help you. I recommend developing a strong understanding of the HTM sequence memory algorithm as it will likely be reused in other layers, and after that I recommend learning about the new sensory-motor concept we posted earlier this year.
Jeff
Thanks for the detailed reply. Definitely understand that this is an evolving theory. I’m really just now digging into these documents in depth, so thanks for the links (I am aware of the new sensory-motor concept discussion, but haven’t gotten into some of the documents linked in that thread yet). Hoping to get a firmer grasp on what you all have currently, so I can follow and apply the updates as they emerge (I’m still pretty new to HTM as you can tell).
My original question was really more about what it means for one layer to project to another layer… seems like from your reply that in your current believe, layer 4 does not project to layer 3. Instead these layers both process the same inputs, just in a different way – 4 getting more temporal input from 6a versus other cells in 4. Is that a good definition of one layer projecting to another? (i.e. one layer’s distal dendrites receiving most of their input from another layer)
Paul,
Here is what is known about the anatomy of L4 and L3 connections.
L4 and lower L3 (L3b) both receive the same input onto their proximal dendrites. This input comes from relay cells in the thalamus. These axons are considered “drivers” as they make strong connections and under the right conditions are able to make the cells fire.
These same axons from the thalamus are known to connect to inhibitory cells in L4 and L3. In HTM theory these connections are key to the spatial pooler. They force all the cells in a mini-column to learn the same feed forward patterns.
L4 cells project to proximal dendrites on upper L3 (L3a) and L2 cells. We view L2 as a “pooling” layer, meaning L2 cells will stay on over multiple patterns in L4 and L3b. In HTM theory, L2 represents the “object” where L4 represents the individual locations on the object.
L2 cells make synapses onto apical dendrites from L4 and L3a. (Some L4 cells are called spiny stellate cells and they lack apical dendrites, but I believe the evidence is that L2 cells still connect to these cells on distal dendrites, potentially serving the same purpose.) These feedback connections are onto basal distal or apical dendrites which are not sufficient to cause a cell to fire. In HTM theory when a cell recognizes a pattern on its distal and apical dendrites it will become depolarized, which leads to very sparse activation in context.
So yes, L4 and L3 both receive FF input and yes L4 connects to L3.
I’ve been meaning to make a connection diagram to get my head strait based on my understanding as well. Jeff cleared up a lot of confusion so here’s my attempt a connection diagram based on his comments:
Some of this diagram was taken from the sensory-motor post:
@jhawkins I strongly suspect that Layers 6a and 6b are doing this conversion. These layers are heavily interconnected with their equivalent layers in the what and where regions. For example in a what region, 6a gets input from layer 6 in the equivalent where region. The cells in L6a then project to L4 (they represent 65% of the input to L4). L5 in the what region projects to L6b which projects to L6 in the where region. This is exactly the kind of connections we need to go back and forth between locations in object space and locations in body space. I am optimistic that we can figure out exactly how L6 is doing these conversions and also how L4 and L5 interact with L6.
The theorization of specific layer functions and how they connect to each other is definitely very interesting. However, it does diverge a bit from my original question (of course my own fault for attaching an outdated/incomplete connection diagram and quoting Jeff from a couple years ago).
I would like to understand the question of what specifically is meant by a layer projecting to another layer. Does this refer to any of the following, a combination of them, or something else?
The original input (after spatial pooling) being fed forward (so that the same columns would be active in both layers)
The output of one layer becoming the input for another layer (a second spatial pooling step treating output of one layer like sensory input)
One layer providing temporal input for another layer, driving its predictive states (i.e. one layer generating distal connections with cells in another layer, versus with cells in its own layer)
Intentionally non-specific due to either gaps in current theory or complexity too long or irrelevant to explain in whatever context the term is used.
I’m very curious about Paul’s questions as well. L3b spatial pools based on its proximal dendrite inputs to select active columns. Therefore, do all other layers use these same active columns? Or rather does each layer individually spatial pool based on their own proximal dendrite inputs? The latter makes sense to me arithmetically, but the former makes sense to me biologically hence the confusion.
