I sketched a SDR encoder for arbitrary vectors, inspired by the fly hash concept.
With the slight difference that the output SDR is “trained” for uniform probability of its bits firing on the initial training data set.
I will explain later what that “learning” is.
I used it to encode MNIST digits in SDRs and then tested the resulting SDRs with:
- simple classification with SDRClassifier as in the mnist.py htm.core example
- a makeshift SDR Associative Memory (pretty much what mraptor requests here)which I’ll discuss in another topic.
First how FHEncoder works:
There is an imput data vector (of floats) which we want to project into a SDR with certain SIZE (e.g. 2000) and number of ON bits further refered as LENGTH (e.g. 100).
Sparsity is simple LENGTH / SIZE
There is a parameter called “Spread” which is further explained.
Let’s see how it encodes flattened (784 long vectors) MNIST digits
- each input point (MNIST pixel) is randomly projected to a fixed number of output points (e.g. 200 out of 2000) with weight 1. All other (2000-200=1800) output points have weight 0 attached to that input pixel. This 200 value is the “spread” parameter
- the output pixels scoring top LENGTH sums will put 1 all remaining will put 0 in the encoded SDR. So its a K-Winner Takes All with K equal to the LENGTH parameter.
The above summarizes pretty much the Fly Hash algorithm without learning.
The learning step adds further 2000 adjustable “output weights” to the results of step 1 above, that are kind of “multiplicative biases” . The algorithm increases or decreases all non-zero weights attached to the output point in such a way that its output value is ~=1 in 2% of the cases and less in the rest.
So the so called “learning” for every single output pixel works as following:
- from (e.g.) 10000 training MNIST digits select its 2% highest output values.
- calculate the mean of these 2% best results.
- pick a multiplicative weight equal to the inverse (1/y) of the above mean (1/mean)
So what the “learning” does is adjust the outputs before K-WTA in a manner that each output has a 2% chance to “fire” on a similar dataset as the training one.
That’s why I used quotes for learning because it doesn’t actually learns anything, it can be regarded more as a statistical balancing operation which tries to avoid some output neurons firing too often or all the time and others seldom firing or not firing at all
The logic behind this output weight adjustment was simply the idea that a neuron that fires always or never has very little value from a learning perspective.
When I combined the above FHEncoder with a SDR Classifier, I noticed the results on classifying MNIST digits where in the 95% accuracy, only 0.1-0.4% behind the results when using a trained SP.
In order to provide “equal grounds” for the classifier the FHEncoder generated SDRs were adjusted to match the ones outputed by Spatial Pooler in mnist.py:
- SDR size of 6240
- sparsity of ~7.7% (*)
(*) Note: I know 7.7% is an odd value. The untrained SP initial sparsity is the “canonical” 5%, but after training I found it to increase ~7.7% . I don’t know why this is so or what it means.
TLDR
I managed to put them in a github repository.
The relevant files are:
- fly_hash_encoder.py - the encoder itself.
- htm_mnist.py - a slightly modified mnist.py example. The changes are explained in the beginning of the .py file.
- flyh_classifier.py - the FHEncoder test with same train/test data and matching SDR size / sparsity.
- load_mnist_data.py, mnist_data.npz - a local fast loader for mnist digits in compressed numpy saved format. The sklearn online loader (originally used by htm.core’s mnist.py) was quite slow in my corner of the internet.