(Yet another paradigm, not exactly about a presentation. Thanks for posting this amazing idea of discussing deep generative models, BTW!)
An interesting fact popped up in my head.
There’s something called VQ-VAE(Vector Quantised-Variational AutoEncoder).
Their key idea is to have some sort of code-book and only use the vectors present in it for the latent representation.
To this end, the algorithm picks the closest vector in the code-book from the original encoded vector.
(by argmin… Do you see the resemblance to k-winner-take-all? )
I was thinking about it and I couldn’t help but feel something very familiar… then it struck me:
This vector quantization concept is very analogous to the SDR representation except it’s for deep learning.
They have some very similar properties as a consequence:
- VQ only allows combinations of the code-book vectors. → SDR only allows specific combinations of sparsely active mini-columns.
- Which forces it to define very limited (but robust) manifold compared to the original (N-D or N-bit) space.
- Both representations make the inputs to be almost always transformed to settle to a point on the said manifolds which makes them really robust.
- With the well-defined mainfolds, it’s very efficient to learn something useful really fast.
- TM can even do few-shot learning!
- VQ enables fast convergence.
- The list goes on…
Another interesting fact:
VQ suffers from different code-book vectors representing almost the same thing and having dead vectors meaning they hardly get used.
And this is extremely similar to what SP suffers from as well!
SP alleviates this problem by boosting and the DL community has come up with something to resolve this as well.
Maybe boosting would also work for VQ as well?
I want to hear what the HTM community has to say!