Biologically plausible boosting


#1

Having been following HTM progress over many years, it strikes me that the encoder stage - generating that initial SDR to feed into the system (e.g. a GPS coordinate encoder) remains something of a black art. In the sense it must be addressed on a case by case basis, there is no generic auto-encoder that will be robust to a wide variety of inputs.

I was reading this paper recently, http://brain.mpg.de/fileadmin/user_upload/Documents/Papers/Papers_Laurent/Assisi_et_al_NN07.pdf, which covers how neural inhibition facilitates generation of a SDR. It maintains sparsity, while effectively preserving the overlap (similarity) between representations. It seems like an elegant solution as well, that could well play a key role in any biologically plausible encoder mechanism.

While you can’t use this in its basic form for auto-encoding (of say GPS coordinates), I’m wondering if it could serve as the basis for a generic encoder mechanism? I haven’t read any Numenta/HTM material touching on this topic of inhibition modulating sparsity, which kind of surprised me, as it seems like a pretty important concept.

Taking the concept of grid field (location) cells, could you combine these cells into an encoder to produce a location SDR? From what I understand grid cells could basically create a fixed precision location SDR output already, but combining it with this mechanism could provide a variable precision SDR based on the training set.


#2

See “boosting” or Homeostatic Regulation of Neuronal Excitability


#3

We can also control sparsity during spatial pooling by activating a higher / lower percentage of minicolumns, which is caused by whatever inhibition is causing minicolumns themselves.


#4

Apologies for the delayed reply! Yes I had seen the material on boosting.

It was my understanding (and please correct me if I’m wrong) boosting was developed as a top-down response to the problem of sparsity and regulating over-active cells. By that I mean, some columns were observed to be activated too frequently, or not enough, and some mechanism was required to achieve homeostatic regulation. Boosting was proposed and developed as a solution to that issue, but lacks the bottom-up fundamentals that are desirable in a model that seeks to mimic its biological counterpart.

It is my understanding the boost level is a modifier per column, that is applied regardless of what SDR might be active at the time (in other words, it ignores surrounding context). Is it correct to say the boost level is defined by a single scalar multiplier per minicolumn, rather than expressed as a function of inhibition from connections? If my understanding of this is wrong, you can probably skip the rest of the post :). If however this is how boosting is currently implemented, it doesn’t sound biologically plausible to have a column regulated up or down in this kind of global/absolute sense, regardless of what other columns are active. I suspect such a model would have too many unwanted side effects on previously learned data.

To give an example by way of conjecture… When we learn a second language, say an English speaker learning to hear/understand a tonal language, there is some level of linguistic overlap (SDR overlap). The subject will have to learn how to recognize those new tones, which will share column activity with their native English representations. If those shared columns were simply down-regulated with no care for context when deemed over-active, this sounds like something that would degrade or diminish the ability to understand English. Since as you down-regulate some of those shared columns, it necessarily results in some decay of the English SDR representations. In practice, people learning second languages appear to suffer no loss of ability to understand their native tongue, there doesn’t seem to be any of this decay whatsoever. Perhaps there’s so much redundancy, the effects can’t be observed, but I feel that unlikely.

Contrast that to the modulation through inhibition. You learn a new tonal language, and the new SDR for tones conditionally (contextually) suppresses those overactive shared columns. Remove the tones, and you’re back to English with no loss of fidelity. Having the boost level not fixed, but a function of the proximal (active) columns, means it can be selectively suppressed without deleterious side effects on previously learned data.

When I have more time (could be in the far future at this rate, but I hope to get there!), I might try to demonstrate this empirically. I would seek to create a metric for how much previously learnt SDRs were decayed (essentially forgotten) from boosting, then see if switching it to an inhibition modulating model made any difference.


#5

Yes I believe so.

What we are trying to model is observed in biology. How we are attempting to model it is not biologically plausible, you are correct. That being said, I’m not sure how important it is to implement in a biologically plausible fashion as long as the functionality of the theory is working.

Your speculation is thought-provoking, but we have not run into a problem with our boosting implementation yet. If we do, I’ll remember this thread. :slight_smile: