Does generalization happen at SP column-level?

Hi,

Does generalization happen at the SP column level? How do you think it happens and does it converge? Why do you think it doesn’t happen and does this mean it is overfitting on a certain set of bits?

1 Like

First we need to define “generalize”. It means that the neurons have the same response to different but related inputs.

How are the inputs related to each other?

Inputs can be related by being the same input with small variations. The spatial pooler can generalize over inputs which are corrupted by random noise.

The inputs can be related to each other by being adjacent in time, aka they’re elements of a sequence. I’ve previously posted about how the SP can be modified to generalize over sequences by using a low-pass-filter.

Inputs can be related by abstract concepts, and it is not understood how that works. For example: apples and oranges are both edible.

1 Like

Thanks, do you have observations of its generalization power at a column and algorithmic level? For example, how does it algorithmically prefer bits that it sees? I should’ve been more specific when I said at a column level, I meant algorithmically.

The reason for asking this is I want to get some views about whether the sequence of training data matters a lot in SP training (online) just like in DL systems. I think it matters a lot than DL’s because in DL weights are regulated, in SP, AFAICT a column can easily overfit on a a set of bits.

Agree and I think this is the answer to the questions mentioned initially,

Does generalization happen at the SP column level? How do you think it happens and does it converge?

1 Like

Yes the order matters. For DL the training samples are usually shuffled. Shuffling also helps with training the SP but obviously can’t be applied to the TM.

The issue is that seeing a very repetitive input (like staring at a wall) will cause the synapse permanences to saturate. It will treat all of the samples as valid, even though it really has only seen one example of the thing its looking at.

A biological solution is for the synapse learning to only happen when either the pre or postsynapse changes its activity. For more about this approach see:

2 Likes

Or you can turn to the biology and see what solution is used there: Habituation.
The response is reduced in cases of repeating input until this input is ignored.

"The broad ubiquity of habituation across all biologic phyla has resulted in it being called “the simplest, most universal form of learning…as fundamental a characteristic of life as DNA”

2 Likes

I think biological solution is to not stare at a wall, any longer than it takes to fit its representation. That’s why we have saccades, etc. The brain will treat a sample as the same if its egocentric location is the same. And it will update that location in proportion to its own motor feedback + detected external motion.

1 Like

Well no, you don’t always have a control over your sensory inputs, and sometimes you might need to intentionally look at repetitive inputs to solve a task, so the synapses should not catastrophically fail… For example the sound of a river or warerfall might be constantly in the background and you should ignore it instead of associating it with the current events.

2 Likes

My point is that sample sameness is determined mainly by its egocentric location, computed from motor feedback + detected motion of the sample. That also goes for your waterfall example: its assigned source location doesn’t change.

Input in the same computed location is not likely to change much, aside from illumination that also changes depending on relative location of a light source. That’s the idea behind object permanence. And when the same object does go through significant change of its properties or motion vectors, it attracts corresponding amount of attention.

1 Like

Do you have a concrete example of this? I know it’s intuitive but algorithmically it can be inhibited by other columns that have more connected synapses.

That seems to be application of anti-Hebbian learning: if source location is recognized to be the same as the location already in memory, then it will be suppressed in proportion to the detail of that memory.