Some updates on my progress!
Experimented with online learning:
Using the output of the spatial pooler fed into a logistic classifier results in 80.60% validation accuracy on MNIST dataset for 1024 cells. Takes around ~8 minutes per epoch.
Experimented with mini-batch learning:
Experimented with mini-batch-based learning for spatial pooler. Basically we run the SP algorithm on a batch of inputs and then update the permanence all at once. This allows parallel training (takes advantage of GPU), making computations much faster (brings each epoch down significantly from 8 minutes to ~30 seconds). I used batch-size of 32, 2048 output cells and a learning rate (AKA delta permanence) of 0.01 with 2% sparsity.
It's a slight deviation from the HTM theory's proposed algorithm but my brief experiments show no degradation in final accuracy. Interestingly, it achieves a 95% validation accuracy (98.37% training accuracy).
A simple linear classifier reaches 88% accuracy based on http://yann.lecun.com/exdb/mnist/. This probably suggests the SP creates a better representation of data compared to a classifier trained directly on the raw data. But I think more work is required to verify my results. Suggestions are welcome!