As I understand HTM theory, if an SDR input arrives and a cell within that input was predicted from the previous input it will depolarize quickly causing an inhibition of the other non-predicted cells in its column. However, there are far fewer inhibitory cells vs excitatory in the cortex.
My questions are:
Does inhibition require a similar set (number) of synapses to achieve the unique properties of excitatory input? In other words, a key premise of HTM is that different inputs can be differentiated because their overlap score is unlikely to be high provided there are enough synapses to go around. But with fewer synapses available on the inhibitory side this seems to suggest a bottleneck issue.
I’ve read that inhibitory synapses tend to target dendritic shafts rather than spine heads. Based from on-path and off-path inhibition it seems likely that inhibitory synapses provide a more general functionality then the site specificity of excitatory synapses (perhaps answering my first question), but if that’s true it still places the network prohibitively at the discretion of only a few inhibitory cells. In this scenario, how does the system retain insensitivity to background noise?
It only needs to inhibit the column of cells whenever any of those cells was predicted and the column is active.
For example, imagine that predicted cells fire slightly earlier than the others when the column is active. It could activate an inhibitory neuron which inhibits the whole column, before the other cells fire. The inhibitory neuron just needs to inhibit all the cells in the column, so it doesn’t need many synapses. It could just use 1 synapse for each cell in the minicolumn, if it inhibits it very strongly.
There are a lot of types of inhibitory interneurons, so the ones responsible for this predicted minicolumn thing might be different.
It isn’t so few inhibitory cells overall. Maybe just 1 inhibitory cell does this for each minicolumn, but there are a lot of minicolumns so it’s fine if some have noise at a given instant. Maybe each inhibitory cell does this for multiple columns somehow, but then each column has multiple inhibitory cells for this so it’s fine if some inhibitory cells have noise.
Thank you for your response Casey. I think I’m still missing something though.
I agree the theory suggests an inhibitory cell can shut down an entire mini-column, but this only exacerbates the issue of a small amount of (inhibitory) noise could have a remarkably strong downstream effect. It seems like an additional mechanism is required to ensure robustness on the inhibitory side. Otherwise, a relatively few number of inhibitory cells have a lot of power over the entire network or is there some other compensatory mechanism?
It’s true that noise in one of those inhibitory cells could have a big effect compared to noise in an excitatory cell, but temporal memory (sequence memory) has enough noise tolerance. The standard number of minicolumns in temporal memory is 2048 with 40 on. If a few of those 40 have noise, it won’t really matter because it probably won’t change what is above or below threshold.
Maybe inhibitory cells have more noise than that, but you could add some redundancy, like multiple inhibitory cells per column or a little time to average out noisy inhibition.
Here’s a hypothetical example about noise tolerance. It might be best to read/watch other things on noise tolerance in HTM.
There are 40 active minicolumns, with 1 inhibitory interneuron each. 20 were predicted and those should be inhibited, resulting in only 1 pyramidal excitatory cell firing in each of those minicolumns. 20 weren’t predicted, so those should have all of their cells fire.
5 of the predicted active minicolumns have noise in their interneurons, and all of their cells fire. This doesn’t really matter because the single-cells-which-would’ve-fired-without-the-noise still fire. The pattern is still recognized, for example leading to the same predictions (activating distal dendritic segments). The extra firing cells could hypothetically cause some extra predictions, but not too many because this is noise so there’s no learned pattern.
5 of the non-predicted active minicolumns have noise in their interneurons, so each has only 1 cell fire. This doesn’t really matter because sequence memory has noise tolerance. For example, a dendritic segment could have 30 synapses, 20 of those synapses on, and a threshold of 10. 5 missing won’t make a difference.