Cerebellar internal models
Internal models in neuroscience are the neural networks
that simulate input–output relationships of some
processes, such as controlled objects in movement
execution. ‘‘Internal” implies that the neural network is
within the brain or the cerebellum. ‘‘Model” implies that
the neural network simulates a target dynamical
process. When humans move their bodies quickly, the
necessary computation is too difficult to be solved by
simple feedback controllers alone, and internal models
are necessary (Gomi and Kawato, 1996). Here, we
include both forward and inverse (see below) models as
possibilities. There is always feedback control, but except
very basic feedback control, model-predictive controllers
or precomputed ballistic movements include some forms
of internal models. When humans grow their bodies and
manipulate different objects, their dynamics change dras-
tically; thus, internal models cannot be genetically prepro-
grammed or fixed, and must be acquired through learning.
Ito (1970) proposed that internal models are acquired by
learning in the cerebellum (Table 1; Fig. 3A, B). Neither
Marr nor Albus mention internal models around 1970,
but Albus (1975) later proposed an artificial neural net-
work model called CMAC (cerebellar model articulation
controller), and one of its possible applications was to
learn inverse dynamics models of robots.
Internal models are classified as either forward or
inverse. Forward models possess the same input–
output direction as controlled objects and simulate their
dynamics. For example, controlled objects such as
eyeballs or arms receive motor commands and
generate movement trajectories, that can be
represented as sensory feedback about the executed
trajectories. Forward models receive copies of motor
commands (corollary discharge or efference copy) and
predict movement trajectories or sensory feedback
(sensory consequence). In control engineering and
robotics, forward models have been and still are just
called ‘‘internal models.” Jordan and Rumelhart (1992)
coined this term to discriminate them from inverse mod-
els, and this terminology was soon adopted by cognitive
science and neuroscience. On the other hand, inverse
models simulate inverted input–output relationships of
controlled objects. In a sense, inverse models provide
inverse functions of modeled dynamical systems. Inverse
models of controlled objects can receive desired trajecto-
ries as inputs and can compute necessary motor com-
mands to realize the desired trajectories.
[…]
In another interpretation based
on the forward model, part of the cerebellar
hemisphere provides a forward model of the controlled
object. It computes a predicted trajectory while receiving
the efference copy of the motor command. The difference
of the realized trajectory and the predicted trajectory pro-
vides a sensory prediction error that can be used as an
error signal to train the forward model. This forward model can be used as an essential element for internal feedback control, bypassing the long-loop through the external world, reducing feedback delays and increasing control performance.
Neural Evidence of the Cerebellum as a State Predictor
Abstract
We here provide neural evidence that the cerebellar circuit can predict future inputs from present outputs, a hallmark of an internal forward model. Recent computational studies hypothesize that the cerebellum performs state prediction known as a forward model. To test the forward-model hypothesis, we analyzed activities of 94 mossy fibers (inputs to the cerebellar cortex), 83 Purkinje cells (output from the cerebellar cortex to dentate nucleus), and 73 dentate nucleus cells (cerebellar output) in the cerebro-cerebellum, all recorded from a monkey performing step-tracking movements of the right wrist. We found that the firing rates of one population could be reconstructed as a weighted linear sum of those of preceding populations. We then went on to investigate if the current outputs of the cerebellum (dentate cells) could predict the future inputs of the cerebellum (mossy fibers). The firing rates of mossy fibers at time t + t1 could be well reconstructed from as a weighted sum of firing rates of dentate cells at time t, thereby proving that the dentate activities contained predictive information about the future inputs. The average goodness-of-fit (R2) decreased moderately from 0.89 to 0.86 when t1 was increased from 20 to 100 ms, hence indicating that the prediction is able to compensate the latency of sensory feedback. The linear equations derived from the firing rates resembled those of a predictor known as Kalman filter composed of prediction and filtering steps. In summary, our analysis of cerebellar activities supports the forward-model hypothesis of the cerebellum.
It seems like everyone who studies the cerebellum is obsessed with inverse models and reinforcement learning models, and has overlooked forward models. Forward models are really useful, you just need to know how to use them!
Here is a procedure for getting motor commands out of a forward model:
Decide on what sensory input you want to control.
For example: the angles of a jointed limb.
Setup a very simple feedback controller. motor_commands = setpoint - sensory_input
Run the forward model to see what the motor commands are actually going to do. run_model(sensory_input, motor_commands) -> predicted_sensory_input
Re-evaluate the feedback controller using the predicted sensory inputs.
The new motor commands are adjustments to the original commands, add them to the original commands.
Repeat steps 3 & 4 several times, and then return the motor commands
I made a proof-of-concept implementation of this idea, and used it to solve the cart-pole balancing challenge. It solves the task on the first try.
The feedback controller can be very simple and easy to tune because the cerebellum is doing all of the heavy lifting. Without a cerebellum this would need a much more elaborate and finely tuned feedback controller.
def get_motor_command(position, angle, angular_velocity):
setpoint = 0
a = 1.0
p = .01
w = .01
return setpoint - (a * angle + p * position + w * angular_velocity)
This controller on its own can not balance the cart-pole; those linear coefficients need to be tuned according to the physical properties of the cart and the pole and the motor. The cerebellum allows the controller to see the future and the controller uses this capability to optimize its actions. Every time you repeat steps 3 & 4 of the algorithm you get a chance to correct any mistakes that the controller made. This should iteratively approach the correct solution.
My answer to “what is the cerebellum doing” fits in one sentence:
I think the cerebellum sits there, watching ≈300,000 different signals in the brain—including signals within the brain, signals going down to the periphery, and signals coming back up—and it learns to preemptively send those same signals itself, but a fraction of a second earlier.
I don’t think my proposal counts as either “forward model” or “inverse model”—or at least, it seems pretty different from what people are usually talking about when they use those terms.
I claim that my proposal is consistent with everything known about the cerebellum. (Happy for criticism! Come at me!)
I don’t like dmac’s proposal here (if I understand it) because I’m not sure how those models get into the cerebellum. In particular, if they’re learned, then I don’t understand what’s the ground-truth that they’re learned from. Or if they’re in the genome, that doesn’t seem compatible with cerebellar anatomy, and doesn’t seem compatible with the cerebellum being helpful for controlling a jet-ski or controlling a six-legged videogame character or lots of other things that were not in the ancestral environment. Right? Or sorry if I’m misunderstanding
The cerebellum learns by watching sequences of sensory inputs, and predicting the next element of the sequence. The ground truth is one time-step into the future.
Really? Your comments upthread (“Run the forward model to see what the motor commands are actually going to do”) imply that a “forward model” ought to have the property that it’s possible to query it with hypotheticals—like, if I did X, what would happen?, but without actually doing X—and change what you do based on the answers. My proposal does not have that property, right?
I think it may be both. Originally cerebellum evolved to learn from / predict sensory inputs, but neocortex mostly took that over. So in mammals cerebellum mostly switched to predicting and replacing cortical outputs from sensory inputs, after the cortex does all the heavy lifting. Deep hierarchical loops in the cortex are very slow, cerebellum is flat and fast.
Yes, but I think the cerebellum keeps doing what it always did, and is responsible for most of the ‘heavy lifting’ on short time scales, as well as quite a bit of ‘automatic’ behaviour.
Cortex brings two things to the party: (1) heavy duty computational ability, building and running sophisticated models of the world (2) attention, focus, working memory. It can do wonderful things, but on just a few things at a time. Your cortex cannot keep you alive.
Cerebellar functions are entirely evolved, with complex capabilities built up over millions of years. Given the dynamic chaotic of what it does, we are unlikely to understand it deeply.
Cortical functions are novel: computational in character, based on a small-ish replicating unit of columns. We may yet get to figure out how that works, which for me is the exciting bit.
Cerebellum can’t learn from the cortex in the animals that don’t have any. Heavy lifting is hierarchical learning, that’s why it’s slow. Automatic must be easy, else it’s not automatic. That’s why cerebellum evolved long before the cortex: fast and shallow learning is relatively simple. BTW, there’s been a lot of talk about “system 1 and system 2”, but I never heard anyone mapping them to cerebellum and cortex, respectively. I think that’s a perfect fit.
That’s funny, I thought it’s obvious that cortex is hugely more complex than cerebellum, especially in combination with thalamus. Those “chaotic” people always get it backwards :).
My interest throughout is understanding the one bit of the brain that might really help AI, namely the mammalian cortex. That’s where Numenta has made some progress, and the regularity of the columns suggests a computational architecture with a small-ish basic unit. The cortex has zillions of units, but it’s complexity of function, not complexity of architecture.
So I don’t really have much interest in the cerebellum, except as a target for cortex output.
But no, I don’t agree learning has to be fast. The so-called ‘muscle memory’ learned over years by athletes and manual workers I would have thought most likely resides in the cerebellum and below.
I always thought of the cerebellum as a very large SDM that hashes the sensory input and motor command together and outputs a random modulatory signal for a set of loosely coupled oacilators that behave like those resevoir networks and can be decoded by the cortex as both a timecode for muscle activations or as a sensory preddiction.
Whatever floats your boat. But all animals are capable of a range of fine movements based on complex sensory inputs that provide action or feedback on timescales too short for the cortex. Many of those are learned (like riding a bicycle or juggling). Where do you think those are located?
What animals are those with a cerebellum but no cortex and how are they relevant to mammalian cerebellum function which evolved in tandem with a cortex for at least a couple hundred million years?
PS regarding complexity - I would not downplay it, considering most of our neurons are in the cerebellum.
Cortical function can’t be understood without thalamus, which is very complex. I am talking about phylogenetic complexity, not a learned one.
Cortical columns are actually quite fuzzy and controversial in neuroscience, not some easily defined unit. In functional terms, think they are best understood as connectivity-based clusters. On a neural level, it’s simply a predisposition of pyramidal cell axons to fork-out laterally, and some innate mechanisms of their interaction with interneurons and basal dendrites. The shape and functionality of resulting columns can vary widely.
In cerebellum it’s a lot clearer, I think each cluster is represented by one of those huge Purkinjie cells: Purkinje cell - Wikipedia. Hebbian learning in a single cell would make it a centroid-based cluster, a lot more primitive than connectivity-based cluster.
The problem with connectivity clustering is that it’s hard to parallelize, you need to merge them hierarchically. Hierarchical connections in the cortex are mediated by the thalamus, so I think that’s where this merging is done, especially in mediodorsal thalamus.
I think any learning is compression, with a significant lossless component, else it’s at best a recording.
And any compression is some form of clustering: the clusters are compressed / generalized representations of their elements.
Oh, like fish for example. It’s quite recognizable across species, same basic architecture: Cerebellum - Wikipedia
the spinal cord can respond to stimulus on its own, thats the point of hitting the knee with that hammer to test the spinal cord response.
aside from that. honestly I dont know how thats even possible but I reckon I watched in a lecture that some output axons from cerebellum skip the cortex and go directly to lower motor neurons while some axons in cortex skip lower motor neurons and connect to muscle fibers directly.
storytime:
once I was playing with a bottlecap trowing it at the wall and trying to catch it when it bounces. I had gooten pretty good at controlling the direction of the bounce by setting the initial angle and rotation of the cap, then I decided to throw it as fast as I could, I was obviously going to miss it but when I started the motion to catch it, my arm suddently felt as if it was duplicated, one arm was still while the other was in motion, both arms duplicates passed through eachother as if both werent real, then suddently the moving arm finished moving and disappeared and the one that remained was holding the cap in its closed hand.
Re the cortex, I can’t agree. The striking thing about columns is that it is a repeating architectural unit, and that across a range of species more columns means more intelligent (along with a few other factors). Apes and humans have evolved relatively quickly, and at least one of the reasons is likely that it’s easier and quicker to evolve by replicating something you’ve already got, than to add function to an existing complex mechanism.
And the Numenta breakthrough around SDRs and sequence memory is consistent with a compact set of algorithms and data structures applicable across a wide range of higher brain function. If you don’t think so, then why are you here?
