Hey everybody,
I have a question about the desired sparsity.
Well, typically we want 2%.
Now, in the SP algorithm (lets assume we have global inhibition) we have 2% of the columns active.
So lets look at an example:
total_numer_of_colmns = 2048
k = 40
cell_per_column = 4
–> total_number_of_cells = 2048 * 4 = 8192
Now we have 40 active columns in the layer. In the TM we activate at least 1 cell per column. Let’s assume for this example, that we have EXACTLY one cell per active column.
this would result in
number_of_active_cells = 40
Because we have a total number of cell of 8192, the overall sparsity of cells is:
overall_sparsity_of_cells = 0.48%
Now I’m not sure: In different papers it says, that the total number of active cells shouldn’t be to low, in order to recognize patterns reliably.
So my questions:
Q1 : Is the required total number of active cells meant in absolute term (so like 40 active cells instad of 2%)?
Q2: is there any disadvantage in just activating e.g. 0.48% of the cells? Or is it the opposite: Is it good for the false positive rate if we have a overall activity less than 2%?
Q3: How would the system be effected if I would choose k = 163 (163 is 2% of the total number of cells).
I would really appreciate some help.
Thank you all very much.
Helena
1 Like
We always say shoot for 2%, but of course like everything in the brain, it’s much more complicated than that. Remember that the SP and TM are defined as different algorithms, but they are actually occurring in the same cell population. You are not counting the boosting that is constantly causing minicolumns to fire for new sequences that may never be finished. That could add a significant amount of firing.
Before I answer your questions above, what HTM implementation are you using? I don’t recognize a parameter called total_numer_of_colmns in NuPIC.
Hey Matt,
Sorry for confusion 
. The names of the given parameters are just chosen randomly.
They do not refer to any implementation.
I just tried to summarize the TM algorithm… and then wondered about the high sparity of neuronal activity if we have 40 active columns.
In HTM, there are a few ways to control sparsity, and you can also measure sparsity in different ways. It could mean the k in k-winners, meaning how many minicolumns will win a competition. But by increasing the number of minicolumns without changing k you will also have sparser representations.
How many cells per minicolum will also affect sparsity, depending on how you measure it. Thinking from the TM standpoint, from the perspective of a neuron, sparsity in distal segments is important. How many segments are formed per neuron, how many synapses per segment, and how many cells per minicolumn can all affect the sparsity of the temporal representation, so it’s not quite so simple as thinking about it at 2%.
1 Like
I’d say both the relative sparsity (2%) and the absolute number of active columns (40) are important.
Keeping the relative sparsity low allows the system to distinguish between different inputs spatial traits. If each input activated 50% of columns instead of 2%, there would be so much overlap between all different inputs it would be difficult to tell the difference between many of them.
Keeping the absolute number of active columns high enough limits the representational burden on each column, since the spatial traits of the input are spread over these columns. So if there were only say 10 columns instead of 40, that’d place a big burden on each column to represent a LOT of spatial traits. Likewise if there were only say 8 cells per column instead of 32, that could put a big burden on each cell to represent a LOT of temporal traits.
I think its invalid to think about sparsity of the total number of cells in the system. Think about the proportion of active columns (like 40 of 2048 = 2%) – and how many cells per column (like 32). If all cells in a column active at once that’s a BURST - where the column activates without expectation. If some subset of cells active those are the cells which were anticipating their own activation through their distal segments.
Again don’t think of setting sparsity in terms of total number of cells, think of setting the sparsity on a column level (like 40 of 2048, etc). The number of cells which activate in a column is totally determined by the TM algorithm – which cells have active distal segment(s) at the current tilmestep.
2 Likes