Good article.
Although, just a quick note about your article. I don’t believe that cells physically close to each other needs to be semantically similar. That’s a major constraint and would make any embedding very difficult since I don’t think semantics can really be fit onto a 2D plane. I also don’t think this is how biology does it. I usually imagine an SDR encoding as a finite set. An active bit means an element is present in the set. This also means that it doesn’t matter where the cell is that activates this bit. They are all completely unordered with no topology. (I would change this assumption though when working with spatial data like images).
Now onto your claim that the performance will be worse because of loss of information. I’m not sure your intuition here is correct.
It’s true that information will be lost, but I don’t think this is such a bad thing. If you have an embedding of 1000s of points in a high-dimensional vector space, arranging all those points so that they are only near to other semantically similar points is a big challenge. In a tight high-dimensional box like this, every point is related to every other point, even when they practically shouldn’t be. The difference between semantically similar words and dissimilar words could be a normalized distance of 0.45 and 0.55. That’s bad when you’re trying to organize and represent your knowledge by semantic similarity.
If you convert this dense embedding to a sparse SDR encoding, it’s like you’re taking the points out of the tight high-dimensional box and unspooling them onto a flat surface as a sparse (possibly unconnected) graph. All those points that were previously at distance 0.5 or higher (if this is a threshold we choose) would have no edge between them on this graph. That cuts number of edges from O(n^2) to something like O(k*n) where k is the average number of similar points per point. You would calibrate this threshold based on your dense-to-sparse conversion process.
As part of your dense-to-sparse conversion, a graph can be created from this encoding with the following rules: an edge exists between two points if their SDR encoding shares at least one bit in common. If an edge does not exist between two points, they share no bit in common. The similarity characterized by an edge is computed by the number of common bits between the two node’s encodings. Then you can visualize your semantic embedding as a graph (by embedding it AGAIN in a 2D space for plotting).
So you see, you do throw away information, but it’s information that adds noise and confusion to any classification algorithm. It also eliminates things like catastrophic forgetting if you keep adding new data points to the representation space, since you aren’t worried about trying to prevent creation of unintended relations to existing points.
See my old post when I try to visualize this similarity space for different types of encodings.