SP bit allocation and permanence distribution - check please?

Engineering Implementations
1

Just a quick check before I plow too far ahead.

For each minicolumn in the SP, I’m randomly assigning it to cover 85% of the input space. So let’s say the input space is 100 bits, I’d randomly assign 85 bits to each column. So it might look like:

bitIndex [
69  87  68  11  18  44  19  55  34  12   8  63   6  20  83  98  32  82   7  47  38  24  90  95  60  48  23  51 49  54  59  16  46   4  43  88  62  81   9  94  93  10 100  74  53  97  86  27   3  29  39  15   2  30  73  37 92  79   1  70  84  76  52  61  78  96  40  64  13  28  22  57  75  45  58  99  33  67  35  72  56  17  50  42 26
]

Now each bit gets provided with a connection strength - a probability(?) based on (in the BHTMS video) the Bates distribution. R doesn’t have a Bates distribution package so for the moment I’m using a normalised set of random values from the Gaussian distribution. I trust that this will provide a similar result but I lose spread. I’m not sure what spread does in the scheme of things or how I can create it.

A sample (set at 0.5 for the mean) could look like:

connection strength [
0.60103513 0.57026347 0.43002539 0.11646626 0.50108318 0.46466792 0.87351216 0.39327734 1.00000000 0.56354117 0.19212880 0.28902620 
0.43751157 0.45969732 0.02251633 0.49232039 0.11694557 0.68648624 
0.52249915 0.33810388 0.37202503 0.15917496 0.53915104 0.24458409 0.17315487 0.53013682 0.71182700 0.72204489 0.73182648 0.55848882 0.46004000 0.45927843 0.64988051 0.66639401 0.32214958 0.84431090 0.28412525 0.66865256 0.39163562 0.32780309 0.84169011 0.49824317 0.57577230 0.62324383 0.71459386 0.14388605 0.43048777 0.36589480 0.49305956 0.41296791 0.37830490 0.66346054 0.53839175 0.62023405 0.52430233 0.78970842 0.75555416 0.50514248 0.38341092 0.00000000 0.16030620 0.71222004 0.32365226 0.21540433 0.07266623 0.70164424 0.49336306 0.26870267 0.69585617 0.66229086 0.48304038 0.60066287 0.41025932 0.24654319 0.25674045 0.31491709 0.74763822 0.49061837 0.56659119 0.62640320 0.38515504 0.24975090 0.10471607 0.41423905 0.39617021
]

So I map the index for the bits available to the column to the index for the connection strength.

Is the above correct and can someone explain how spread affects the connections - and a way to recreate / use it when creating a normal distribution?

TIA.

Rob

2 Likes
2

I’d recommend you check out the source code for SP:

Specifically this part where the SP columns perms are initialized:

for columnIndex in xrange(numColumns):
  potential = self._mapPotential(columnIndex)
  self._potentialPools.replace(columnIndex, potential.nonzero()[0])
  perm = self._initPermanence(potential, initConnectedPct)
  self._updatePermanencesForColumn(perm, columnIndex, raisePerm=True)

It looks like all permanence values for each 1 SP column are uniform, so I think your use of normally distributed values is a departure from this. What I’m not totally sure about is what the perms are initialized to. There is a hard-coded parameter in the constructor to the ‘SpatialPooler’ class called ‘initConnectedPct’, though I’m not sure if this sets the init values for all perm values. Maybe you could verify @rhyolight or @subutai?

3

Thanks Sam.

The initial values are what I’m referring to. In the BHTMS video on initial values RandomBates is used

1 Like
5

Yes, I diverged from the NuPIC code to establish the initial connections in a normal distribution around a center point. The most important thing is that perms are initially close to the connection threshold. The normal distribution not necessary, but makes more sense to me. I think @alavin suggested it to me years ago.

1 Like
6

Thanks Matt. Has there been any experimentation done in regards to performance / learning for an SP configured as such compared with a set value? It would be interesting to see if it made a difference.

7

It’s been a while and I’m out of touch with HTM, but maybe this helps: The Bates distribution is a uniform distribution of random variables on the unit interval. Intuitively it makes sense to use a Gaussian distribution (with mean at the threshold) s.t. most of the density is around the that threshold. The implication being most permanence values won’t need to fluctuate much to converge, while still having flexibility to explore the space further away from the threshold; Bates may put too much density far away from the threshold.

4 Likes
8

Thanks for that clarification Alex.
So to distribute percentages I can generate a set of percentage values (0 - 1) and then use the Gaussian density function to allocate them across the bit indexes. I think this will give the same result as using a random Bates function except as you say - with a density closer to the required threshold. My confusion was in trying to constrain random Gaussian values to between 0 & 1 which didn’t make a lot of sense.