7 Likes
Very nice, thank you for sharing.
You’re welcome.
1 Like
I was just planning a discussion thread for possible mechanisms of layer 5 motor output as I am currently focused at that part of the agent. Great timing. Hopefully these will open up a good discussion.
1 Like
I am glad you shared these. Below I am going to pick apart some arguments for the sake of coming up with an actual workflow in terms of an HTM architecture. Naturally I will have to reduce the complex functioning of layer 5 into some variation of an HTM layer, bear with me.
First things first; what really causes layer 5 firing? (Proximal input in terms of HTM)
since layer 5 receives a direct sensory input from thalamus which evokes firing
Layer 5 even fires when there is no sensory stimulation, suggesting its firing does not code sensory items
So does sensory input modulate or drive layer 5?
Or does layer 5 sometimes get driven by thalamic input and sometimes by some other source (possibly L6)? More importantly, is there a chance that L5 is driven by motor input (maybe included in thalamic input) rather than sensory input? Or does L5 have a constant excitation baseline independent of the input as hinted below?
L5 cells have noisy spontaneous input, which adds a degree of randomized firing. This random activity has been proposed to contribute to experimental behavior. Further exaggerating their firing patterns compared to other layers, L5 cells are by far the most bursty
I am very happy to read this. Randomized firing without even requiring input seems necessary whenever I tinker on the agent architecture. This is not how a normal HTM layer behaves and I think layer 5 needs its own variation after reading this.
It does not need to detect the information on the sensory surface, only which sensory surfaces are stimulated. Even if it did detect that, it would still appear as if it were not particularly selective…
This seems to be a running theme for your layer 5 hypothesis. I always thought TM is the phase where details are added to the rough sketch (minicolumn activation) of SP. The point above implies proximal input (rough sketch) may really be sensory thalamic input. However, I need it to be motor thalamic input.
L5 projections to subcortical sensory structures are predictive because L5 in at least some regions has longer latency than all subcortical sensory structures, so it must change the sensory response before L5 itself responds.
Can you elaborate on this, I could not quite understand? Also, what are the subcortical sensory structures in this case? (like LGN?)
Even though L5 TT doesn’t project to L5 ST, their proximity suggests shared inhibitory cells, leading to linked oscillations, linked competition, and linked minicolumnar states.
Is it reasonable to start with a single HTM layer emulating both thick tufted and slender tufted cells? Or do you think separate layers are a must depending on differing functionality?
1 Like
I don’t see what you see as contradictory. Many sources contributing to an effect certainly does not make our task easy, sorting them out… but it’s not implausible either ?
1 Like
You are right, I reworded that part. A lot of sources driving layer 5 doesn’t really play out well with HTM layers, hence my stance.
2 Likes
The direction I was going with that is based on the idea you mentioned in a post that L5 has noisy activity to generate experimental behavior, or something along those lines (no citations yet, sorry!)
I think that must extend to a sensory role, seeing as L5 is noisy in sensory regions, too. It’s true that those sensory regions have motor outputs, but ~half of L5 cells in those regions are still TT cells, whereas you’d expect them to be sparsely distributed if they only had a motor role.
So to apply noisy firing to sensory things, I don’t see the noise being too useful for representation, and it’s not useful for generating behavior at any given moment, but it is still useful for learning sensory things like just for behavior. For example, let’s say L5 learns precise timing in sensory processing. Then the noise could contribute to testing slightly different timings. The same goes for processing distance, or any other continuous variable. I don’t have any ideas about the specifics, I just imagine learning mechanisms would cause it to latch onto a randomly good response.
If L5 represents where things are and when things happen, it’s in a pretty good position to generate behavior, so it might as well serve that sensory role too.
To summarize the upcoming rambling, I also think L5 generates predictions, not the same kind as in sequence memory though because it operates on precise timing scales. That ties into behavior in a vague way. I’m more thinking about a sensory role. It’s based on bursts appending spikes to the end of the event which caused the burst, and spike timing dependent plasticity.
(some rambling)
I also think L5 generates predictions, not the same kind like in sequence memory. When a cell bursts, that appends some spikes after the event which caused the burst, so it can associate that even to things maybe 40 ms later. If something occurs coincident with or even a bit after the first spike, it will undergo LTD postsynaptically from the bursting cell (because some spikes occur afterwards), assuming normal spike timing dependent plasticity. I think there must be actively represented predictions to generate behavior, even just because the time delay would otherwise be too great. Presaccadic predictive activity (presaccadic remapping) occurs in cells which project to the superior colliculus, so TT cells, and the presaccadic activity in that layer but not L4 includes a weakening of the response to the current sensory input, suggesting L5 TT has a role in active prediction. I also think active prediction is required for thought, but who knows. Anyway, the point I got a bit distracted from is, noise can help test predictions, both by generating random ones and adjusting the predictive response a bit. These effects could occur postsynaptically in the targets of L5 TT cells, or just in lateral connections between TT cells.
I was looking at my notes for burst learning rules and this effect might occur presynaptically to a bursting cell. Now that I think about it, that needs to happen for the predictive remapping signal from L5 TT to mesh with a role in firing-represented prediction. The source doesn’t clearly show the compartment on the cell where this occurs. When a cell bursts, it dissociates from presynaptic inputs which occur between 125 ms and 25 ms after the burst begins (5). It also dissociates from inputs 200 ms before to 200 ms after burst onset, but -125 to +25 ms is the window of greatest synaptic depression and there’s single firing mode plasticity to consider which might bring it to synaptic input sufficient to burst sometimes, but not too often, leading to a kind of equilibrium of plasticity caused by the two firing modes leading to opposite plasticity.
I don’t know about TT cells because I’ve done basically no research on L6, but the article you linked is about L6 CT activating L5a pyramidal cells, so that signal from L6 to L5 is to slender tufted cells. There’s still probably an input from L6 to L5 TT since that seems to be part of some ideas at Numenta.
There are probably multiple driving inputs. Things get a lot more confusing for me when I try to figure out the exact flow of signals through the circuit because each layer or sublayer is connected to a bunch of others and the single cell processes are hard to research and not well understood, so it’s easier for me to think of each layer as serving certain roles than performing a particular step in an algorithm.
As far as I know, motor thalamus connects to the same layers as sensory thalamus. But L5 thalamic input is generally not found by studies, even though it exists in V1 (1), barrel cortex (2 and many other sources), and A1 (3), so probably everywhere, and this can drive responses. I suspect this isn’t commonly accepted because the other sources of input drown out this pathway and because L5 is thought of as an output.
I got pretty frustrated when I was researching behavior because I couldn’t find clear behavioral pathways and didn’t make progress, so my opinion is that corticostriatal cells are the decision making cells. L5 TT cells can project to the basal ganglia, but they project to pretty much everything subcortex and that projection varies a lot between species (I think I’ve lost the source, but this one found only a minor projection in one species (4)). L5 TT cells drive behavior, but that could just be a reaction along the lines of, "there’s an object, so look at it/grab it etc. I have no clue how to think about proprioception and that sort of motor-like sensory thing, but maybe the same idea applies. Maybe sensory input is the same as motor input if both are about locations.
The idea that layer 5 processes which parts of the sensor are receiving input was was the starting point for the rest of those ideas. Sort of like processing entire macrocolumn states (binary input or no input), although maybe on a finer scale than that.
It’s sort of sequence processing, but maybe not in the same way that each state in temporal memory chains into the next to represent the sequence so far. I don’t see it as a replacement for temporal memory, just a step beforehand but still after spatial pooling generates the initial minicolumn states. Beyond that, it gets chaotic. What happens when propagating oscillations collide and how does that depend on the sequence of contact? It seems like they would produce chaotic patterns. I think I’m missing something and I need to figure out something more specific.
I don’t think proximal sensory input is incompatible with proximal motor input. How does the motor map in motor cortex line up with the idea of macrocolumns in sensory cortex?
Maybe each macrocolumn or the points on a finer map correspond to points in a motor space, so it represents a change in location of the sensors moved by the behavior. That would imply L5 represents a coordinate transform or movement. Jeff Hawkins says they think L5 represents displacements, I think in allocentric space: What spaces does L6a and L6b represent exactly relative to objects? Are these layers connected in any meaningful way? - #4 by jhawkins. Maybe L5 in egocentric regions does that, and L5 in allocentric regions represents the location itself because it is already relative to things since it’s allocentric.
Basically, 99% of what I write is spit-balling thoughts that pop into my head. Generally, cortex responds to the sensory input after subcortical structures, especially higher regions in the cortex and/or later parts of the cortical response, where latencies can be hundreds of millseconds. If subcortical structures are to utilize that information, the cortex needs to send predictions about that information ahead of time.
Here’s a more concrete example. In presaccadic predictive remapping, neurons start responding to the sensory input before a saccade to it. Some articles claim that, without shifting RFs ahead of time, you would see the blurred visual input caused by the saccade, so the brain skips ahead. There are some other reasons to do so, like generating rapid sequences of saccades where there isn’t a good visual input between saccades.
It’s a similar idea. There’s latency from subcortical structures to cortical structures (and latency between everything) so, at least when precise timing matters, predictive signals are required to contribute to the very initial response, or even a large part of the response for some long latency cortical responses.
I don’t have any particular structures in mind right now. Any subcortical sensory processing, which probably occurs in motor structures, too. For example, perhaps the superficial layers of superior colliculus, inferior colliculus, or pretectum. I don’t think LGN is a possibility for modulation or enhancement by L5.
After writing that, I read that they receive inputs from different types of basket cells somewhere. It’s hard to tell whether they share competition, though. Also, ST and TT don’t seem to compete with each other via martinotti cells (7), and they might form separate minicolumns (8). That latter study is based on synchronization and ST cells have longer latency sensory responses than TT cells, at least in barrel cortex, so they could still share minicolumns.
I think they’re as different as L4 and L2/3. They have a lot of similarities, so a single layer would probably work for some things and help figure out why there are separate layers despite the similarities.
[1] Three Types of Cortical Layer 5 Neurons That Differ in Brain-wide Connectivity and Function (Euiseok J. Kim, Ashley L. Juavinett, Espoir M. Kyubwa, Matthew W. Jacobs, and Edward M. Callaway, 2015) https://www.cell.com/neuron/fulltext/S0896-6273(15)00981-2
[2] Deep Cortical Layers are Activated Directly by Thalamus (Christine M. Constantinople and Randy M. Bruno, 2014) Deep Cortical Layers are Activated Directly by Thalamus - PMC
[3] Laminar Structure of Spontaneous and Sensory-Evoked Population Activity in Auditory Cortex (Shuzo Sakata and Kenneth D. Harris, 2009) https://www.cell.com/neuron/pdf/S0896-6273(09)00720-X.pdf?code=cell-site
[4] Corticostriatal cells in comparison with pyramidal tract neurons: contrasting properties in the behaving monkey (Bauswein et al., 1989) [PDF] Corticostriatal cells in comparison with pyramidal tract neurons: contrasting properties in the behaving monkey | Semantic Scholar
[5] Firing Mode-Dependent Synaptic Plasticity in Rat Neocortical Pyramidal Neurons (Barbara Birtoli and Daniel Ulrich, 2004) Firing Mode-Dependent Synaptic Plasticity in Rat Neocortical Pyramidal Neurons | Journal of Neuroscience
[6] Surround Integration Organizes a Spatial Map during Active Sensation (Scott R. Pluta, Evan H. Lyall, Greg I. Telian, Elena Ryapolova-Webb, and Hillel Adesnik, 2017)
[7] Disynaptic Inhibition between Neocortical Pyramidal Cells Mediated by Martinotti Cells (Gilad Silberberg and Henry Markram, 2007) https://www.cell.com/neuron/fulltext/S0896-6273(07)00111-0
[8] Lattice system of functionally distinct cell types in the neocortex (Hisato Maruoka, Nao Nakagawa, Shun Tsuruno, Seiichiro Sakai, Taisuke Yoneda, and Toshihiko Hosoya, 2017) http://science.sciencemag.org/content/358/6363/610
2 Likes
The interactions between effects might only be complicated when viewed as a circuit from input to output. That’s probably necessary to a degree, but by trying to figure out what it does rather than how it does things, each of those contributing sources narrows down the possible explanations rather than complicating things. Like how the object layer resolves ambiguity.
Are you familiar with the deepleabra model?
Starting with the leabra model.
It is the most plausible model I have seen to offer something that looks a lot like back-prop in the cortex.
This has been advanced to the deepleabra model that fits well with the known properties of both the L6 layer and the Pulvinar. It combines the “slow to respond” L6 and thalamic drive to form a nice predictive model.
https://grey.colorado.edu/emergent/index.php/DeepLeabra
BTW: As far as it goes - the feedback path in regards to the sensory stream is feed-forward when it terminates in the motor drivers.
I know that it won’t be popular to say this here in the heart of HTM land but I see the possibility of three mechanisms all at work at the same time:
- HTM style feed-forward predictive
- Deepleabra feed-back predictive
- L2/3 hex-grid forming sparse-discipline / inter-area communication cells.
- As described in the deepleabra model - the pulvinar is part of the L6 predictive model and a control channel to spread activation between related maps.
- The thalamus projections from the HTM cells should work well with the RAC to predict activity and gate in the predicted sensory input. Note the two states here - bursting and gating. This is compatible with the HTM model where simple predictions fire one cell and act as a thalamus tonic. Bursting in the column could trigger the same excited state which would gate much more of this surprising sensory stream to be learned.
I see a very powerful synergy with these parts working together. Feel free to tell me why this is all wrong.
3 Likes
I have to go to bed so I’ll just give a quick reply for now.
No, but it looks interesting. It will probably take me a while to understand, but it seems to incorporate a lot of features of neurons which aren’t explained by or required by HTM.
I’m not sure what exact feedback and feedforward paths you are referring to. Do you mean that feedback signals activate cells in layer 5 which project subcortically? Some sort of interaction between sensory cortices and motor cortices?
I’ll need to read about deepleabra and your posts about forming grids, and do some thinking.
So long as those are translatable to HTM speak or at least are related, which I’m sure they are, I wouldn’t worry too much about saying that. Talking about plain deep learning would be different, of course. (I’m kidding.)
If it won’t be obvious after some reading, can you give me some examples of that synergy? Do those three fulfill different roles and/or interact in a particular way? I’ll probably have questions about those later.
I’m not going to have a reason to tell you that it’s all wrong. I’ll probably disagree a lot, but just for the purpose of generating ideas and figuring out how to think about the problems being solved.
I guess it is early for algorithmic takeaways but I am kind of forced to make some during modeling. That’s why I pushed towards some mechanistic speculations with those remarks. Algorithmic speculations of more people would certainly help.
The architecture in the thesis functioned similar to this. If ganglia and other layers depolarized layer 5 cells at the same time, those cells would be both predictions and motor commands via association with motor neurons. However I think this was limiting due to a variety of reasons.
The layer 5 activations constantly remapped to motor activity in that architecture. Do you think that layer 5 post synaptic targets change constantly like that? In other words, can the same layer 5 activation change its corresponding motor behavior in time? This presented problems because layer 5 predictions (biased by rewards) are based on the motor associations which weren’t really that stable. I am shooting for fixed connections between layer 5 cells and motor neurons this time. I would assume there are strong connections between spinal cord and layer 5 cells that are not that fragile for the sake of stability. Do you have any comments on this?
The motor behavior should have some randomness to it for exploration but here is the question; is this randomness caused by layer 5 activations or other motor structures? In the architecture, there was already randomness in motor neurons that would fire without L5 influence. Layer 5 activation would then override corresponding motor neuron activations for a favorable outcome. Instead, do you think layer 5 does the exploration here by introducing randomness instead of motor neurons themselves?
1 Like
Thinking from the perspective of a hierarchy, I imagine that you might also want to introduce randomness from the layer to enable exploration at higher levels of abstraction (rather than only at the lowest level).
1 Like
I have been spending some time with tractology - what is hooked to what and which direction those connections are going. There seem to be paths originating in the forebrain hopscotching through the maps eventually ending in the raw sensory inputs. Mostly these paths skip through the lower layers and thalamus. These are what I am calling feedback.
Going the “other direction” there are pathways streaming from raw sensations to the forebrain. These are primarily through the upper layers. I am calling these feedforward.
Within the forebrain, the unfolding motor plan goes from the lower and polar regions of the forebrain and terminate in the motor driver strip. I can see a case to be made to call the actions that unfold to drive internal activity in the brain as a sort of motor activity. These activities do not directly drive muscle fibers but they do drive activities that can eventually be sensed in the temporal pole.
Solid reason for going with randomness on the level of layer 5.
What’s your take on fixed vs associated connections between motor neurons and L5? Your perspective hints at dynamic connections considering hierarchy but if the agent cannot consistently get into a target state signified by L5, the state values computed by RL do not converge properly.
Example Case
Assume L5 activation wants the agent to get into state B from A by activating motor neurons associated with B. But some other transition in the past changed the motor association between L5 neurons representing state B, so the actual motor output fails to take the agent to state B. If A->B was a path to reward, this failure would result in the state value of A dropping because the agent could not reach B as expected. Maybe I should insist on association and work on a better design preventing this.
1 Like
Do you mean they are plastic over LTP timescales or are you talking about the same output generating different behaviors because of other activity?
I think connections to the spinal cord have to be pretty stable, at least past a certain point in development or learning a particular skill. I’m not sure about other things. Maybe what’s stable is which part of the maps they target (motor maps and perhaps sensory maps), so a given neuron generates a particular behavior, but postsynaptic learning could still occur to fine tune timing, for example.
I don’t think L5 TT predictions are biased by reward, at least not as a central role. The way I think of it, L5 TT cells just generate a bunch of options, and L5 ST and basal ganglia pick out of those options as well as subcortically generated options, in part by influencing that L5 TT activity. I don’t know much about motor cortices, but at least in barrel cortex, higher order thalamus drives L5 slender tufted cells, which are much more corticostriatal than L5 TT cells in the small number of regions I’ve studied. Since the basal ganglia only control the cortex through the thalamus as far as I know, that means ST cells are better positioned for being biased by reward in some sort of feedback loop. There might be a loop from L5 ST to basal ganglia to thalamus back to the same ST cells. Using the signals from L5 TT to the same thalamus that projects to ST cells and how basal ganglia gates the thalamus, L5 ST could then cause L5 TT cells to burst since they target L5 TT cells on the distal apical dendrite, or maybe directly cause firing through proximal connections.
Another possibility is that both L5 ST and TT are biased by reward, but TT cells are more concerned with reacting to sensory stimuli, including in complex ways because it needs to handle things that sometimes occur when the same behavioral plan is being executed, whereas ST cells are more concerned with what it will do next after reacting to the current sensory input.
I think at least one group of cells, whether or not it’s L5 TT, needs to have unstable motor results so it can learn locally in the cortex. Maybe that’s why L5 has TT and ST. TT cells would operate on the level of representing sensory inputs and direct behavioral responses to sensory inputs, with noise to produce variable responses to explore relatively fine scale changes in behavior. They would optimize things like timing and positioning based on some reward perhaps, but also perhaps predictability of the behavioral results. It has to deal with intrinsic noise a bit, but I’m more thinking that it explores different precise, small variations in behavior so it finds behaviors that have the same results in many slightly different scenarios.
That circuit could perhaps operate independently from ST cells, helping it evolve. Then added on top of that, ST cells would control TT cells to produce the behavioral results that TT cells would otherwise only produce by reacting to things. For example, ST cells might form a circuit with the basal ganglia and thalamus which selects TT sensory reactions. By adding an additional source of depolarization to TT cells, which are already usually near threshold, the ST circuit would cause TT cells to move beyond only reacting in the sense of, “hey look, it’s shiny! Grab it!” and causing them to be able to generate behaviors which are sensory reactions only in the sense that those behaviors are possible in that sensory scenario. The ST-BG-thalamus circuit would generate more original behavior, and also guide TT cells to follow a particular sequence. Since ST cells would be more closely linked with goal-oriented reward (as opposed to sensory prediction-oriented basal ganglia circuits, if that’s a thing), they wouldn’t care as much about the current sensory input and would instead do planning.
Layer 5 cells have low thresholds, I’ve read. I’m not sure what that means, and it might just be an average. Since they have low thresholds even when not behaving, I think they generate noisy firing because background activity-caused noise peaks above threshold much more if the threshold is reduced a bit.
Regardless, I think you’re right that subcortical motor neurons need some noise. Maybe not the ones that are direct motor outputs, but instead the ones the basal ganglia could control. It has to work without the cortex. There’s still the advantage that Paul Lamb brought up of cortical generated noise allowing experimentation with abstract things
There’s a bit of a contradiction in my opinions. L5 TT cells seem to be a source of noise, but at least at face value, the brain doesn’t want noisy cells which directly control muscles because it doesn’t want to experiment when it knows what to do. Maybe noise in precise timing is okay though, or maybe the noise averages out to the same muscle or central pattern generator or whatnot. I’m not too comfortable with L5 TT cells being corticospinal cells, but they are. If I were comfortable with them being corticospinal cells, I would be uncomfortable with L5 TT cells being noisy and receiving directly driving sensory input.
Some more thoughts a bit later:
What if L5 represents the state it will bring the agent to? If the basal ganglia were to make decisions based on that represented state, it could avoid bringing itself to an unexpected state.
2 Likes
Does L5 produce feedback via the thalamus? That would be pretty interesting. Even if that’s not the case, could you point me in the direction of some good articles about those pathways? It gets pretty overwhelming with all of the names and I haven’t encountered any good summaries.
Maybe the entire L5 feedforward pathway is actually basically feedback since it’s via thalamus which targets L1. I’m not sure how that meshes with L5 also sending signals to L4 via thalamus, but maybe that’s actually not from L5 since I haven’t really looked into that. I don’t think the signals from L5 to L4 via thalamus could be modulatory because they cause driver-like responses in L4 in one article, though.
I want L5 to be the source of general intelligence. My only idea about that is, motor = actively represented predictions = representing things based on what they predict = concepts = intelligence.
2 Likes
Start here:
3 Likes
I did some more writing in hypotheses doc. I wrote some sections I was planning, but I also got a bit trapped into thinking about grid cells because some stuff didn’t seem like it would work well.
The mechanisms are mostly kinda not supported by articles, but L5 could represent allocentric locations, generate behavior by the same mechanisms it uses for correcting the allocentric representation for that movement, and cells which directly generate behavior could still represent and respond to sensory things separate from generating behavior. It’s about 2.5 pages starting on page 8.
1 Like