Inhibition Mechanisms in Spatial Pooling

Education Getting Started
1

Hi all,
I’m a newbie in HTM and spartial pooling. As far as I know, there’re two Inhibition mechanisms in Spartial pooling: Global and Local Column Inhibition. I have a question that is there any other inhibition mechanism that can be used in Spartial Pooling? Thank you in advanced!

2

Inhibition occurs because only the top k minicolumns are activated during spatial pooling. We call this a competition between all the minicolumns in a neighborhood. If the neighborhoods are many and distributed throughout the layer, this is local inhibition. It takes more compute, is more complex, and assumes topology in the input data. If there is only one neighborhood, that means all the minicolumns compete against each other, and this is global inhibition.

Your question doesn’t quite make sense to me, but you can change the size of the column neighborhoods to get a spectrum of inhibitory behavior between global and extremely local.

2 Likes
3

Hi rhyolight,
thank you for your reply, do you have any documentation which describes in detail how local inhibition can choose the winner column? Thank you in advanced.

4

I guess this paper will help you, actually I didn’t read it yet but I think it’s very interesting, it talks about spatial pooler.
https://sci-hub.tw/10.1109/tetci.2018.2850314

5

There is also the Numenta’s biological and machine intelligence documentation (BAMI). It’s got a detailed explanation of the Spatial Pooler’s algorithm including it inhibition phase.

6
2 Likes
7

Thank you all, I’m reading the pseudo code of inhibition:

for c in columns
    minLocalActivity = kthScore(neighbors(c), numActiveColumnsPerInhArea)
    
    if overlap(c) > stimulusThreshold and overlap(c) ≥ minLocalActivity then
       
      activeColumns(t).append(c)

with the explanation:

“The inhibition logic will ensure that at most
numActiveColumnsPerInhArea columns become active in each local inhibition area. For example, if
numActiveColumnsPerInhArea is 10, a column will be a winner if it has a non-zero overlap and its overlap score
ranks 10th or higher among the columns within its inhibition radius.”

However, I still have some question:

  1. How the parameter “numActiveColumnsPerInhArea” is calculated in each iteration loop or it is chosen by default.

  2. I don’t really understand the meaning of the line “minLocalActivity = kthScore(neighbors©, numActiveColumnsPerInhArea)”

For example, I have the set of overlap {1,2,3,5,10,12,1,4,9,7} and the InhibitionRadius is 2, at column 0 what would be the value of minLocalActivity calculated?

I’m sorry for my dummy questions, but I really want to understand the process clearly. Thank you in advanced.

8

numActiveColumnsPerInhArea is a manually set model parameter. We almost always set it to 2% of the total column count (if 2048 minicolumns, it is typically 40).

Watch this video about topology, it explains the minicolumn competition a bit better. It makes more sense when you think about topology, which requires local inhibition.

9

hi rhyolight,

so is it true to say that if we use the local inhibition, the number of active columns would be much higher than the number of active columns when we apply global inhibition?

10

Not necessarily.

Assume a pool of 10000 units.

I could use “global” inhibition and set the winners to 2%, or any 200 units responding.

I could set 10 neighborhoods, each covering 1000 units. If I set those each with 2% response rate I will get 20 responding in each area for a total of 200 units (20 units x 10 areas) but with a distribution distributed equally over the 10 local regions.

The choice of architecture is somewhat dictated by what problem you are trying to solve and how your problem representation maps to the structure.

For larger a larger system that mimics the cortex the local neighborhood is about 300 to 500 µm in diameter for columns with a 30 µm pitch, or about 225 mini-columns with roughly 6 units responding. This gives a 3% sparsity.

1 Like
11

I’m sorry but this field is not clear for me. For example if we have 10000 units, we would have 10000 neighborhood if I understand correctly. for example, I have 10 units input overlap is {0,1,2,3,4,5,6,7,8,9} and the inhibition-radius is 2, so the 1st neighborhood is {8,9,0,1,2}, second is {9,0,1,2,3}… and the last neighborhood would be {7,8,9,0,1}. If I’m wrong, please correct me. Thank you in advanced.

12

In reflecting on this you may wish to contemplate the difference between the input field and the output or responding units.

Can I suggest that you look at this thread first and see if it helps you with your question:

As I try to understand these things I look to the biology that is being emulated for enlightenment. In this case, as you read this keep in mind that the inhibitory interneurons are spread throughout the cortex and there is a balance between column activations and firing of these inhibitory units. A winning column will activate the inhibitory interneurons which will tend to suppress any additional activity in the nearby neighborhood. This is the basis for the sparsity enforcing inhibition in the HTM theory.

This is emulated (crudely) by the k-winner function in most HTM implementations.

In the biology, the Chandelier cells appear to be in the correct position to provide this regulatory function.

Another likely candidate for this inhibitory-regulatory function is the Basket cell.

HTM theory postulates that some of this inhibition action is local to a single mini-column (the core of temporal memory theory) and some acts on a larger scale encompassing the size of a macro-column.

I am still reading about inhibitory interneurons and it is hard to say that one of these types is the “only” source of sparsity or if it is some sort of collective activity. BTW: there are other types of inhibitory interneurons other than the two I listed above. Suffice it to say that some sort of inhibitory action is being performed and it results in the observed sparse activation of the mini-columns.

1 Like
13

@nluu When you go from global inhibition to local inhibition, things get exponentially more complicated. Topology is introduced whether you want it or not. Processing is now distributed across the layer. There are a lot more computations to run. But it does not necessarily mean the overall % of active minicolumns in the layer is affected. Each minicolumn now has a neighborhood of minicolumns surrounding it, which are processing some overlapping topological input. If the neighborhoods are consistent and the number of active minicolumns within each neighborhood is the same across the region, it doesn’t matter how big the neighborhoods are, the overall percent of active minicolumns will be similar.

1 Like
14

Hi @rhyolight, I thought that in local inhibition, the inhibition radius is calculated base o the average span of each column for input space, so we can not choose the radius of neighborhood ourselves. So I don’t under stand the term:

15

It kind of reminds me of sudoku.

Let’s say you’re just using a 4x4 square for local inhibition, and you can only have 1 winner cell, so you want a sparsity of 1/16.

If you run this over a 12x12 input, you’d get something like…
image

Each 4x4 region should have 1 and only 1 winner cell, so red cell wins the first 4x4 region, orange wins one cell to the right all the way to 4 cells to the right, then yellow cell wins, and so on. Since each 4x4 region needs 1 winner, we end up with 9 winners over 12x12, which is still a sparsity of 1/16.

1 Like