HTM seeks to model and understand what is going on in your head.
I tossed this drawing together from memory so it is sure to be missing some connections. I skipped the thalamic connections all together as this was already a very complicated drawing. Also not shown are the connections between the limbic system and the stream passing through the brain stem and the cerebellum. These are important but not necessary to illustrate the basic cortical/limbic data flows. The blue maps are cortex composed of HTM processing modules. The orange modules are some of the key sub-cortical structures.
The blue connections in the various sensory streams are essentially in parallel form.
The broad green hub connections connecting the various highest levels in each lobe are also parallel.
We can quibble about details but it is clear that as data course from area to area it stays in a mostly parallel form. The maps of the various sensory fields, while “crazy quilted,” are preserved in the hierarchical processing.
So where does this all this parallel data come together?
First - I coded the data transformation connections in orange; these may not be what you could consider parallel. When you consider the limbic system the connections look different.
The EC/HC complex has some scaling where representations in multiple scales exist, but each maintains topography. The rest of the limbic system does not look anything like the cortex. The processing models look more like the classic Boltzmann network. Or perhaps a Hopfield Network? What is presented to the lower brain “wraps around” the complexes, transforming the topographic representation into a “spherical input” form. The limbic clusters don’t work like the cortex so it should not be surprising that the topological format is transformed as the data is conveyed. The connections between the inputs to the limbic system and outputs to the prefrontal cortex all do some data rearrangement to make the data formats compatible.
The cortex prepossesses the senses into a form compatible with the older brain structures and after the lizard brain does it’s processing it projects commands to the prefrontal cortex to be elaborated into actions. These actions could be directed to further internal or external data gathering or into motor drives that move the body.
You can think of the level of processing of these lower brain structures somewhat like what a moth does in flying up to the light in response to all the cues that signal mating time. In the moth genetics have tuned it to fly up to the moon to mate; genetics did not plan for porch lights. With our big brains these old senses and drives are vastly enhanced. These senses should be better at processing sensory cues and turning those drives into suitable action plans. I call this my dumb boss/smart advisor model.
At the lowest levels to the forebrain, the output fibers don’t project to the body, they project to the temporal lobes to be experienced as “thinking” and “recall.” This is really the same thing as the lower brain structures pointing the eyes though the FEF (frontal Eye Fields) to look at things of interest, but this part is all internal to the brain. These recalled memories are then experienced by the temporal lobe, hippocampus, and related structures in a loop of experience we call consciousness. The blue parts are available to consciousness, the workings of the limbic system are not.
Some of these forebrain activities may result in the selection and production of motor activity - words and actions. These are all stored motor programs that are being called into play, customized by the recalled memories and drives from the limbic system/forebrain. The networks in the various areas settle into states where there is the least “conflict” between the competing activation patterns. Experienced AI researchers will recognize this as a relaxation computing process mediated locally using attractor networks.
This basic layout or something similar should be a path to AI that has what looks like intelligent behavior.
How is the neocortex connected to this older brain strucures? Do these structures connect to the thalamus, or use L4 and L5 as input and output, or do they connect like inter-cortex connections in L2/3? Is it more like driving input, or modulatory input.
Can you predict phenomenons, that are produced when disturbing this system. E.g. when applying a specific neurotransmitter, or having a lesion in a specific area, then …
You are asking about the fundamental operations of the most complicated structure known to mankind. The detailed answers could easily fill volumes of textbooks. I will try to hit the highlights here but but I am warning you ahead of time - while this is a very long post I am leaving very much unsaid and unexplained. Let’s start with your first bulleted item.
The thalamus is broadly interconnected with the cortex at all points. Some have described this as the seventh layer of the cortex but that is much too simple. The connections to the body pass through the thalamus to the cortex. There are connections paths between the cortex and the thalamus. Numenta is starting to work in this area. See this thread for more details.
The thalamus also drives a basic heartbeat function that binds activity patterns together. This post addresses part of your question about layers and the thalamus:
Possibly the best description that I have read on the interactions between the thalamus and the cortex is in this paper. It’s a demanding read but well worth the effort. It addresses more of the question about layers and the thalamus - it describes the predictive mechanism in the feedback direction. This is distinctly different from HTM which describes prediction in the feedforward direction.:
So on your question regarding information vs modulation - all of the above.
I have been addressing various details of how all these bits work together in the forum for a long time. You can take this collection of posts together as loose documentation of how I see these parts working together. These papers start to address your prediction question; since it describes the basic operations and interactions of the parts it is possible to describe what will happen if you interrupt those functions:
This describes the action of the Layer 2/3 in coordinating various cortical areas and recognizing and communicating spatial and temporal pattern recognition. This is where the lateral and inter-area binding occur. This binding can be described as temporal and spatial pooling.
Everything up to this point has been the answer to the thalamus & cortex part of your question
A major sub-cortical structure is the cerebellum
While it is often thought of as connecting movements together to convert the parallel motor commands from the cortex into smooth sequential commands down the spine, it also feeds back into the brain to make our thought coordinated.
How about those other sub-cortical structures?
I refer you to this post. You will have to click-through this to see the nested links.
What are those subcortical structures?
What do they do?
And what are the subcortical connections to the body and its sensors?
Still not convinced that the lizard brain plays any role in us newfangled mammals? You think that the old structures are just relays up to the fancy cortex?
While the workings of the brain require a cortical presence to enter consciousness there are still hints that the lizard brain workings are drivers for our mix of behaviors.
This is an older post but still very relevant to this topic.
GPU is the “mamalian brain” with so much speed and parallelism. It is homogeneously build of many small building blocks (cores) all seemingly implementing a limited set of simple functions - basic matrix operations.
Yet without the slow “reptilian” CPU with its complicated archictecture, much complex functions… it won’t work.
Without an “OS” and “how the GPU should be used program” implemented in the “old brain”, the “modern brain” would go cataleptic.
I’ve been logic-ing my way through it on paper over the past year or so, and it keeps boiling down to the basic point: If SDRs and temporal memory via distal connection activations is the primary algorithm of the cortex, the input/output from that system would also need to be encoded. That’s to say, sensory input, previously predicted sensory input, andmotoroutput from the previous timestep all need to be presented to the cortex as a concatenated input; the brain must be provided a method to correlate its output with a resulting input.
The logical result of the this necessity is that to make a fully intelligent system based on HTM, we need to add a few components, such as an input/output modulator (control the potentiality of a specific output occurring) which itself is also controlled/influenced by feedback from systems throughout the organism such as physical condition, goal progression, and the surprise expressed by the cortex when it bursts. This might also be where we can plug in a “values” module or governor to artificial system.
And based on what is happening inside of us, biologically, it seems we have evolved those mechanisms (minus the values governor) as well as the corresponding connections between the different brain regions to accomplish that.
Where my experiments are going is trying to create a framework for plugging in these “other” pieces into the SP/TM algorithms. I’m glad Numenta is exploring the Thalamus, as that seems to be one of a couple Grand Central Stations of the brain controlling flow in and out of the neocortex, perhaps serving as an attention mechanism based on input from other parts of the ancient brain.
One thing that the paper does not address is exactly the concern you raise - the connections between the sensory and motor system. I see this as a system level process using the association tracts:
And closely related:
I think that the hands-off view of activation in the chosen task also misses one other key aspect of cortical processing.
Recognition of something is an active recall process. I think that response on the local map level is a pattern completion action that is cooperative between connected maps. At a higher/system level I have read several papers that work through the act of conscious recognition as activation in the hippocampus in ~500 ms, followed by a flow of activation back towards the senses over the next ~1500 ms.
All of this is skipped in this paper as the task in this paper is special in that the global workspace is already primed to the task and when the expected stimulus is sensed it gates an already learned task into action. It is the shortest sensory/response loop that the brain normally capable of doing.
Faster (and simpler) response is handled in the spine processors.
I would like to think that it should be possible but - early days.
In many other fields (physics & chemistry for example) all the “easy” things have been discovered; AI is still in its infancy. This is still the wild west where all the good things are still waiting to be discovered.
I think that you need to include the association region (Parietal hub) to do object integration and connections to the frontal lobe. The basket of features from the sensory processing still needs to be collected and turned into hex-code for the temporal lobe to read it.
In the “baby bird” group we surmised that you need to hear and learn speech first, then run the motors to reproduce this internal model with your speech hardware.