I was reading up on the spatial pooler and had some thoughts on how local inhibition would affect the fixed sparsity property of the SP:
One of the core properties of the SP seems to be that it creates fixed-sparsity representations. With global inhibition the SP undoubtedly fulfils this property as it simply achieves s% sparsity by activating the columns with the s% highest input overlaps.
With local inhibition however this doesn’t seem to be the case anymore. For example, consider some column a that is not activated as it is not within the top s% of its neighbourhood. Some other column b within a’s neighbourhood is within the top s% of that neighbourhood, therefore contributing to a not being activated. However there’s no guarantee that b gets activated as b’s neighbourhood might contain other columns with even higher input overlaps, that were not within a’s neighbourhood.
An extreme example of this would be a one-dimensional SP where the input overlaps decrease monotonically from one side to the other. Depending on the hyperparameters, every single column except for a small fraction at the very edge of the SP would be inhibited by their neighbours on one side, leading to an actual sparsity significantly lower than the ‘intended’ sparsity.
I confirmed that this occurs to some extent through NuPIC. I trained an SP with a dataset of around 50k items for one epoch, with columnDimensions=2048, numActiveColumnsPerInhArea=16 and inhibitionRadius=366 (this last one was decided by NuPIC internally). I then fed the same dataset to the SP again. The number of active columns now ranged between 16 (0.78% sparsity) and 48 (2.34% sparsity), which I would not consider a “fixed” sparsity. The Spatial Pooler paper by Cui et al. in contrast shows the sparsity of an SP with local inhibition always being near the target 2%, calling it an “inherent property of the network due to the use of local k-winners-take-all activation rules”.
My questions on this are:
Is what I described above actually the case?
(if so:) Could such fluctuations in SP sparsity pose any practical problems?
(if so:) Has there been any effort in designing some alternative method to pick active columns when local inhibition is active to ensure the target sparsity is adhered to more closely?