@jacobeverist to me the interesting bit must be the principle on which the encoder is built:
In the ideas I’ve been presenting here (and now exploring with @complyue ) the metric of similarity is the number of common contexts between two representations. This can be seen as equivalent to a kind of cross product for vectors. So, contrasting with yours, which is a form of dot product, this might be seen as a form of cross-product.
What is interesting is that a cross-product generates new vectors, so new structure. Compared to a dot product, which only compares what is known.
There are other ways of relating elements to build “encoders” which expand the data rather than just compressing it. For instance, I think this talk by Chris Domas I referenced earlier, describes a few:
Domas has representations which visually distinguish… “locality” was one (Hilbert transformation) or adjacency(?) I think he had which made text visually distinct from… encrypted binary (so he could easily distinguish in computer code where encrypted areas, or computer viruses… might be? So, identifying general properties of viruses, rather than specific viruses as former tools had?)
The common feature of Domas’s representations, and why I liked them, was because they were all “expanding”. They all generated new structure, which was meaningful even though new, because of the way it related elements, instead of just comparing with known structure.
For me the interesting question is whether a representation of a given set of data is finite and compresses the data. Or if it is infinite and expands ever more structure out of the data like mine, or Domas’s.
I think representations which expand ever more structure out of a given set of data are going to be the ones which turn out being central for the study of cognition.
Has that contrast between encodings which compress data, and encodings which expand structure from data, been something you have looked at in your thinking about encoders?