That’s a nice little auto-encoder. It looks like the main differences between your approach and a more traditional auto-encoder are that yours is both shallow and wide. In other words, you are using only one hidden layer, and that layer encodes the latent variables/features. By enforcing k-WTA, you are essentially forcing each image through a bottleneck (similar to traditional auto-encoders), but on effectively different sub-networks.
Are the latent variables scalar values? Or do you force them to 0/1? Are the features at the bottom the result of passing a single latent variable (SDR size of 1) back through the decoder? If that’s the case, then it looks like your network has learned something akin to an orthogonal basis set (eigen-vectors) or perhaps a set of convolution filters.
I force the values to 0/1 but all features are updated simultaneously so they do affect each other in ways I dont completely understand.
I need to include bosting otherwise the features get stuck and all images end up producing the same sdr.
well, about that, it still kinda works if I make it shallow and narrow.
it seems to be learning some kind of on/off gabor filters/wavelets.
I guess that would mean that you have more data being encoded in the weight matrix than you need. The words that I would use to describe it are that the columns (or rows) of your weight matrix are encoding eigenvectors, and your latent variables are the eigenvalues. Once you have a set of eigenvectors that span all (or nearly all) of the space of possible MNIST digits, then the remaining weights are redundant. That’s not to say that additional weights are useless. Often, the redundant vectors will match different images preferentially and allow you to encode them with smaller SDR’s (fewer on-bits).
Since you are forcing your latent variables to 0/1, it would seem that the filters are adjusting to allow you to stack them in such as way as to obtain brighter pixels in regions where they overlap positive weights and darker pixels where they overlap a mix of signed weights, thus allowing them to cancel out.
yea, thats what I think is happening but the particular shapes they choose are always a little funny and seem to have some gravitating effect where they force others to be similar instead of different, If I could get rid of that I think the encoder would have better performance.
run the SDR backwards through the matrix to get a reconstruction estimate of the image.
here-s a multiplication matrix HTM works with binary synapses attached to dendrites attached to cells grouped in columns, you would have a bit of a problem figuring it out, and is much worse than picking “floats” vs “ints” vs “bits”