I have been messing with stacking multiple layers of Spatial Poolers together and see what happens. I originally thought this is a bad idea, but heck, stacked RBMs are a thing. Why not Stacked Spatial Pooler?
The setup is simple.
- Train a stack of N SpatialPoolers with random values between 0~1. Encoded using GridCell
- Encode and send the value 0.5 to the stack of SPs and get the output.
- Encode and send values from 0~1. See how many bits overlap with 0.5.
Since Spatial Pooler is almost a dimension reduction algorithm. I suppose like any dimension reduction algorithm, the more you do reduce, the less of information gets through and thus HTM should have a harder time to distinguish between values as I stack more and more SPs together. So, when I stacked enough SPs together, the plot of overlapping bits should become a flat line.
Here is the source code:
You will need
to run the experiment.
This is a plot of how many bits overlapped when there is no SP. The encoded value goes straight from encoder to comparison.
And with 1 SP. The plot looks more concentrated.
With 2 Spatial Poolers, it gets rectangular-ish
With 4 Spatial Poolers, it gets really like a rectangle
At 8 Spatial Poolers, it becomes an rectangle. Also a funny thing I find is that the base of the plotted shape seems doesn’t widen up. Instead they contracts.
And with 16 SPs. It is the same as with 8 SPs. Nothing changes. Why? Why does stacking SPs together concentrates the activation. This makes no seance mathematically.
I was expecting the plot goes from a bell-shape initially (which I gets right) to a flat line as more and more SP are stacked together. But no, instead the values that can activate the final layer of SP gets more and more concentrated.
Maybe this is how the Neo cortex stabilizes and categorizes data? By using multiple layers of Spatial Poolers? Any comments or theory on why this is happening?