Marcus Lewis frames the problem of knowledge transfer among cortical columns in the Thousand Brains Theory and explores potential solutions. He first explains how, at a high level, Numenta’s model tackles this problem by having cortical columns communicate a description of an object horizontally through lateral connections.
Marcus then explains how this “horizontal description of an object” mechanism suggests a different mindset for the Thousand Brains Theory. Influenced by Douglas Hofstadter’s book “Gödel, Escher, Bach,” he states that a cortical column fundamentally needs to learn to represent descriptions of objects, so that, given a description of a novel object, it can make predictions and recognize that object no matter where the description comes from. Secondly, the cortical column can memorize those descriptions, but this object-memorizing functionality is secondary to being able to describe an object in the first place.
Jeff then dives into what information about the object is needed for columns to communicate and vote on what they’re sensing. He makes the case that columns communicate locally through lateral connections in the neocortex and work in parallel with each other. The team then asks questions and discusses.
Paper about 3 subtypes of pyramidal cell in layer 5 (mentioned around 27:30)
Maybe descriptions of objects (for knowledge transferability) involves something like actionability. By that, I mean things you can do with objects. Perhaps “what can I do with this object?” is more transferable than “what is this object?”. For example, “how can I hold this cup?” says a lot about the cup’s shape.
(You can’t do much with the color of the cup, but the color changes sensorimotor sequences.)
The neuroscience reasons for that suggestion are related to the striatum, the motor / sensory signal sent from the cortex to the thalamus, and higher order thalamus activating the basal dendrites of the small corticostriatal L5 cells.
Around 23:00, Jeff suggests the striatum is responsible for a shared body-centered reference frame, which helps knowledge transfer. As far as I know, the striatum only influences the cortex via the thalamus.
The thalamus has some broad projections to the cortex, which could help transfer knowledge. Some parts of the thalamus are activated by the large cells in layer 5, which also act as the cortex’s motor output. That suggests every column has motor-related info which gets sent to many other columns. For behavior, it makes sense for each column to be asking, “what can I do with this object?”. That info could be how the columns describe objects.
After that suggestion about the striatum, Jeff mentions the 3 types of pyramidal cells in L5. One of those is the kind which projects to other parts of cortex and to striatum. As mentioned above, the striatum influences the cortex via the thalamus. The thalamus also sends a strong input to those pyramidal cells on their basal dendrites, maybe even proximal. Usually, that’d just be a sensory signal, but the cool thing, it’s from the parts of the thalamus activated by the layer 5 motor output cells.
I think your falling into a bias of too closely correlating the pattern a cortical column recognises as to what it means in relation to the whole process. Separate columns can be programmed and used in isolation to play games and recognise patterns we know and can associate with, but you need to abstract your thought away from these pattern biases in isolation as to the representation in the brain.
The pattern / object is a very dispersed collection of micro “temporal fragments” that is by association voted into existence, not necessarily as a single column concept that we would (conveniently) like to exist as a coffee cup. This mostly fits with the thousand brains but I completely disagree with “where” input as the “where” is just one of many associative correlated inputs, not a specific where, it’s just an associative input, one of many. Sensory “Where” (place/grid column associations) is just another associative concept as to the word “shop” or “guess”. Everything is just a concept (usually tagged with a word pattern for communication - “if it is a desirable effect”) and the shear scale of the brain is what we have problems in conceptually dealing with. Same goes for our inability to comprehend space in relation to the nearest star, let alone another galaxy, it’s just the way our brains work from evolution only having to deal with an environment what we can physically move in.
How we go from parallel (occipital/auditory/sensory/etc) to serial (temporal) to parallel (cortex activation wave) to serial (broca) processing internally is something that is biologically awkward to hold in a transitory state…
You can’t magically create an information bus for a particular purpose that is not an associative activated mechanism. Synapses need a trigger to be created and destroyed or inhibited and the reason for explicit information transfer is where I think you have an issue. Dreaming/REM is a separate interesting process that may “share” information between columns but I think it’s more correlated “association leakage” rather than intentional information transfer. Different close associated correlation amplification with intentional purpose ?
My basic thought would be avoid the bias trap of seeking correlation and building a discovery away from what it does. Columns are a data compression object like RLE.
Maybe cortical columns are the equivalent of transistors in a CPU where we don’t really need to know the quantum mechanics of electron tunnelling to understand what a column “can” do or “does” functionally and then re-look at the overall architecture. Building a computer from transistors is far easier than building a computer from quantum electrodynamics ground up as can be seen from the over half century gap in relative progress. True a quantum computer will make us look like cavemen, but we can still use fire in the interim.
The point made at 19:50 between the cerebellum (animal) instinctual pattern activation to the more complex temporal pattern overridable process in the cortex is not to be missed or misunderstood as to existing purely in the cortex. “Instincts” may dissolve/migrate to the cerebellum and this adds sort of an additional layer of data compression in a dispersed memory array of temporal events.
Maybe I’m wrong, and very likely so, but hopefully a good disruptive thought initiator.
Maybe the cortex handles this by remapping. Instead of a reference frame transformation by a single cortical column (to track a sensory patch’s location), maybe it maps the space onto the cortical sheet. That’d allow a cortical column to learn about things which its sensory patch doesn’t sense.
That seems to happen in barrel cortex and with presaccadic remapping e.g. in FEF. I don’t remember whether this was mentioned or was what Marcus was talking about with lateral connections.
