Encoder to skip SP!

Engineering Implementations
1

Using SP is expensive, I’m looking for algorithms to generate suitable SDR’s directly from the Encoder.

Below is the code of one I created. Pls comment on what benefits/drawback you can see.

The normal procedure is to Encode the values and then pass it trough Spatial Pooler,
but it is expensive operation. What if we can encode directly and skip SP.

This is a Naive algorithm I came up to do just that, here is how it works.

The algorithm depends on 3 parameters.

- vsize : the number of bits of the SDP
- spaOnbits : sparsity percent OR number of bits
- olap_nbits : how many bits do neighboring items overlap

First we have to calculate capacity i.e. how many items we can represent.
For example if the SDP is 1000 bits, spa-bits is 20 and neighboring items dont overlap,
then the capacity is 1000/20 = 50 items i.e. we can generate randomly 50 items the
51st will necessarily generate item that will overlap with some other item.

With me so far ?
Now if we allow 10 bits overlap, we can generate 1000/(20-10) = 100 items, before we
generate item that will create item with bigger overlap.

Now that we know the capacity, how do we do it :

We keep a set of all free for use bits. In the beginning its all of the 1 … 1000.
We generate spa-number-of-bits randomly and remove them from the free-set.
Next we pick randomly olap-number of bits from the previous item (in this case the first one) and non-olap-number of bits from the free-set, where :

	self.non_olap = self.spa_nbits - self.olap_nbits

Then we rinse and repeat until we exhaust all the possible items…

Here is a quick test…

In [112]: ne = NaiveSkipEncoder(minimum=0,maximum=100,vsize=1000,spaOnbits=20,olap_nbits=10)                                                                                 

In [111]: np.array([ne.decode(ne.encode(x)) for x in range(0,100)])                                                                                                          
Out[111]: 
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
       41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 56, 57, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81,
       82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])

Also the similarity maps seem to show that similarity is ~preserved.

BTW the Encoder I crated depends on some other libs I use, so if you want to use it you may need to implement those. Most notably the iLexicon which is KV-SDR store.

more examples of the item :

In [112]: ne.lex[3]                                                                                                                                                          
Out[112]: 16:78:95:155:239:277:294:313:336:344:349:353:378:410:507:580:718:757:887:979

In [113]: ne.lex[4]                                                                                                                                                          
Out[113]: 16:123:155:217:239:261:271:283:294:313:353:507:580:589:676:711:872:887:898:979

In [114]: ne.lex[3] // ne.lex[4]                                                                                                                                             
Out[114]: 0.500  <- similarity 

In [115]: ne.lex[3] / ne.lex[4]                                                                                                                                              
Out[115]: 10 <-- overlap

In [117]: ne.lex[3] // ne.lex[5]                                                                                                                                             
Out[117]: 0.250

In [116]: ne.lex[3] // ne.lex[6]                                                                                                                                             
Out[116]: 0.050

and encoded values :

In [119]: ne.encode(55) // ne.encode(54)                                                                                                                                     
Out[119]: 0.500

In [120]: ne.encode(55) // ne.encode(56)                                                                                                                                     
Out[120]: 0.500

In [121]: ne.encode(55) // ne.encode(57)                                                                                                                                     
Out[121]: 0.500

In [122]: ne.encode(55) // ne.encode(58)                                                                                                                                     
Out[122]: 0.250
2 Likes
2

Not sure if this answers your question.

I have been exploring developing a multi-layer encoder model based on Shri Yantra ( https://youtu.be/gTcWL00NUrs ) an abstract form of universe.

The first layer of lotus with 16 petals captures the raw sensory input and some other essential constructs as name and memory, while the next layer of 8 takes it to next level of abstraction with language.

Imagine : i have an image - how would i encode it? With user input, with following steps- Instead of rendering it into Tensor of Colors on 2D axis, one could say break all image into a matrix like “Bing Maps Tile System” (0123) where user has a dial to zoom/scroll in on the object as a best fit (where locus or vector, goes in memory with other relations), assigning a sound, a name, while the encoder has a geometry to mimic the petals to flash if the input contains those individually encoded data points (image related, sound related, text/name related)

Please let me know if this is too abstract.

If discussion around Shri Yantra as a possible encoding model inspires you, please let me know, happy to elaborate it as a separate post.

3

As a newbie I can’t help feel that I would have trouble representing an input comprised of multiple variables like dates, float, categories, etc. into a single SDR representation. If I naively code an input into an SDR I somehow feel it wouldn’t end well even if I can represent each unique input into a unique SDR presentation. Because if many similar inputs has too many column overlaps it would overload the columns which becomes too active and prevent it from learning effectively. If too little overlap between similar inputs then it doesn’t work either. From the HTM school videos I’m guessing (encoder output) → (SP+boosting) would be helpful in creating a more evenly distributed SDR value.

1 Like
4

the problem with SDR and similar schemes like Kanerva BSC hypervectors is not so much representation, but the small range/steps of similarity.

What I mean is if you have 20-40 bits sparsity you normally get max 2-3 levels of separation i.e. ~50%,…30%, ~15%… that’s the nature of SDR, once you start combining them.

In [146]: ne.lex.a // (ne.lex.a * ne.lex.b)                                                                                                                                  
Out[146]: 0.500

In [147]: ne.lex.a // (ne.lex.a * ne.lex.b * ne.lex.c)                                                                                                                       
Out[147]: 0.300

In [148]: ne.lex.a // (ne.lex.a * ne.lex.b * ne.lex.c * ne.lex.d)                                                                                                            
Out[148]: 0.150

In [149]: ne.lex.a // (ne.lex.a * ne.lex.b * ne.lex.c * ne.lex.d * ne.lex.e)                                                                                                 
Out[149]: 0.050

For comparison for dense representations sim-levels is almost infinite …f.e. difference between 0 and 10 can have upto 10-levels of separation/similarity.

(I’m working/thinking about a mix that can get both higher similarity levels + distributed,noise-redundant format)

1 Like
5

You may wish to rethink your representation. You are describing a parallel format with all of the representation at one level.

Think of how you hold a long phone number as a sequence of digits. To hold it in temporary memory you have it stored as “a short song” that I suspect you repeat over and over from the time someone gives it to you until you execute it by dialing.

Consider that TV ads frequently present their telephone number by making it a repeated song to get you to remember it as a kind of jingle.

So the SP coding space is relatively small, with the meat of the memory represented as a distribution through the hierarchy as temporal sequence.

This is the HT of HTM.

3 Likes
6

@Bitking Thanks that makes a lot of sense.

@mraptor Not sure if this is relevant but there was a post about BrainBlocks that briefly touched on how using the encoder input directly without SP to train the TM can still be practical. The author also mentioned about encoding N-dimensional data for N<=40 and how he is working on the theory of Distributed Binary Patterns trying to address issues of understanding binary patterns, SDRs and which encoder to use, etc. Link: Releasing BrainBlocks 0.6: Building ML Applications with HTM-Like Algorithms - #28 by jacobeverist

1 Like