Hi,
I’m opening this discussion for ideas how to test Spatial pooler implementations !
If you remember Matt videos on Encoders. He mentioned 4 rules which must be obeyed for every Encoder.
I was wondering if we can come up with similar succinct rules of how to test SP.
Let me share my current experiment … I’m trying to Encode ‘words’ and pass them trough the Spatial pooler.
After that I’ve build classifier (which is simply an SDR map between SP-SDR and the word before it was encoded. The best-match classification is done based on either overlap OR hamming-distance, I’m testing both ).
What I do now is to run again trough all the “words” and see how well they compare i.e. predicted-word == original-word.
Currently my error rate is 0.6% i.e. 10 out of 1673 are mismatched, which means that the SDR generated by my Spatial pooler are almost prefect for this small set of data.
Any other ideas how to test and compare different SP implementations !??
What do you guys at Numenta do ?
In [2]: wt = WordsTest(data_size=400, osize=2000) #15 chars word
Building charachter map ...
> Spatial pooler 400 => 2000 (iwc:8,owc:42,ss:1)
> Creating Memory 2000x400 : winp:0.021, nudges:5,None
In [3]: wt.init_nlp()
Creating NLP object ...
In [4]: wt.process_file('data/10000_small.txt')
13252 tokens processed
In [5]: wt.train_spooler()
> 1000 words
> 2000 words
> 3000 words
Time taken : 0:01:33.713692
In [7]: wt.train_clf()
Total number of words : 1673
Building word index map ...
Total number of words : 1673
In [8]: wt.test_predictions()
376> head => dead
519> man => men
694> mary => men
813> deep => beer
1152> edge => fade
1450> talk => tall
1467> beat => chat
1469> photography => photograph
1510> walk => wall
1533> sherlock => peculiar
In [9]: 10/1673.
Out[9]: 0.005977286312014346
The number of words is 1673 out of 13252 tokens, because I’m using word lemmas, skipping stop words and dropping anything that is not a word.
But the SP trains on all the words, sentence by sentence going trough the whole document in sequence, not just the set of all the words.
2 Likes
I have been asking myself this question as I prepare visualizations for the SP HTM School videos. I got inspiration from running @victor’s new spatial pooling example code.
The images is creates are quite informative about how well the SP is doing representing the semantics of the input data:
At least you can tell how training an SP improves its performance. I’m also toying around with a visualization that compares the current active columns of the SP with all the previous ones it has produced to ensure they are similar:
TOP SECRET 
TOP SECRET 
2 Likes
Those tests assume that both INPUT and OUTPUT binaries are sparse, right ??
(In my current case the inputs are not sparse 
, only the output )
No, the input binaries do not need to be sparse.
So If I understood it correctly the test is :
- Pick a input-pattern X
- Pass trough SP : X ==> Y
- Add noise : X1 (flip random bit/s)
- Calc overlap : Ox = X & X1
- Pass trough SP : X1 ==> Y1
- Calc overlap : Oy = Y & Y1
- Go to 3
Plot Ox,Oy
It should be as close to right sloped line.
Correct.
You do this before training and after training SP.
I can see intuitively that similar input-overlap, should correspond to similar-output overlap, but still feel abit icky. Is there some merit to my distrust of such a relation ?
Second, this was just single input-pattern. It could be impossible to test all possible input-patterns (too much time consuming). How would approach this i.e. how many patterns to test and which ones ?
The SP example has 2 stages. In the first one we run steps 1 to 7 with an untrained SP, whereas in the second we do it on a trained SP. The purpose of this is to show how the SP behaves when exposed to noisy input. Both trained and untrained SPs exhibit some resilience to noise (as you said: similar input-overlap maps to similar output-overlap). However a trained SP exhibits more resilience which results in the “sigmoid” curve in Fig 4.
Regarding your question on how to test and compare different SP implementations, I think it depends on what do you want to test for. In the example we are testing for noise tolerance, whereas you might want to test for sets of parameters vs accuracy. For that you could try plotting some learning curves that result from using different set of SP parameters or training epochs.
1 Like
