It’s been four years since my Triadic Memory project was discussed here. That topic became the most-replied-to thread on this forum, and I am still incredibly grateful for the discussions we had.
This project introduces a new computational theory of mind grounded in set theory and hyperdimensional computing. I wanted to share it here first because the architecture is inspired by and closely related to the work being done in the HTM community.
The site includes the full theoretical framework and an open-source, standard-C reference implementation of the core algorithm.
I’d love to hear your thoughts, critiques, and feedback.
I asked my grad student because it was a bit TLDR for me. I thought he was generous:
My first read: this is conceptually elegant and potentially useful even if the AGI claim fails. The strongest parts are the set-theoretic framing, the distinction between SDRs and SHRs, the topological associative memory mechanism, and the known/unknown separation as an anti-hallucination property. The weakest parts are likely the large biological extrapolations and the jump from associative-symbolic machinery to full general intelligence.
For our purposes, Sir, the key relevance is obvious: this is very close to an implementable non-GWT, non-transformer cognitive substrate. It gives us a concrete candidate for ETCB-like binding: not a global workspace, not quantum binding, but local discrete associative closure over sparse symbolic/perceptual sets.
I took a first pass over this yesterday, it looks very interesting but I will definitely need to re-read it a few times.
As a casual enthusiast, my initial reaction is that when it gets heavily into the math realm (set theory and graph theory) I started to lose the link back to the biology that was promised in the intro. I understand it’s hard to talk about the physical implementation and the computation model at the same time, but some kind of boxed summary in each section would stop me getting lost in the more abstract.
I look forward to digesting this and eventually will have some real questions!
I am still reading but so far it resonates a lot with what I have come up with over the years. I have some reservations for associative memory approaches to AI but I will wait till I am done before making any judgement
My major problem with using the above approach is that it seems more like a solution to information retrieval than AI.
For AI we need a lot more than that. First we need to learn x to y mapping. Given an input representation the system should be able to learn a way to transform or map x to y. Secondly, the approach should be able to generalize. i.e. predict things by using multi hop reasoning or transformations as opposed to simple pattern matching or lookup.
I am open to discussing more about these concepts if you are interested. I look forward to seeing how far you can take them.
Do you really need an (explicit) mapping. My proposal is a geometric object where the prompt /input, stands in relation to the hidden geometry which stands in relation to the output.
Here the rules of the geometry construct a prior for valid output
You can have simple coarse-fine associative memory lookup.
A coarse binary vector and a fine linear mapping.
A binary vector can easily act as a key in a hashtable and the hashtable return a linear mapping.
The funny thing is that can actually be chained.
If the input is x then b(x) is a function giving the coarse binary vector (approximately where am I) and a linear mapping m (fine lookup) is obtained by a hashtable lookup m=h(b(x)).
Non-linearity in conventional neural networks is quite fine grained. That is not fundamental to its operation though.
Suppose you had 4 weight matrices A,B,C,D. If you just composite them you get the linear mapping (linear associative memory) y=DCBAx.
Since Z=DCBA can be pre-computed you might as well compute y=Zx instead.
Unfortunately then most of the parameters in DCBA are wasted. If you get 2 binary decisions from x: d₁(x), d₂(x) and pair the matrices (A,B) and (C,D) you can use the decisions to decide between A or B, and between C or D.
That gives you 4 possibilities:
y = CAx
y = CBx
y = DAx
y = DBx
That makes full use of all the parameters in the 4 matrices especially if the decisions are about 50:50 for the given inputs. The decisions could be derived from say random projections of the input x or any other reasonable way.
If you had 20 binary decisions you could use those bits to select between 2²⁰ matrices:
y = M₀x
y = M₁x
…
y = M₁₀₄₈₅₇₅x
Or you could choose to select between 20 pairs of matrices (20*2 = 40 matrices.) Obviously doing the composition of matrices (linear mappings, linear associative memories) results in strong compression of the number of parameters needed if the data being mapped will actually factorize nicely, if it will play along with such a compression scheme.
My point is that we need a system that learn a mapping not a way to make an explicit one. I am talking about unsupervised learning. The system would not just need to learn that x maps to y but has to come up with intermediate representations that make the mapping possible and produce generalisation. Generalization here would be the agents ability to infer that a and c are related if it only knew ad → b and cd → b. Note that the inferences also need to be somewhat consistent with reality
Back-propagation will do. You don’t need all the messing around with derivatives, which makes it a lot simpler to work back through the composition of matrices.
I’m not saying it is a good system in itself though. It is quantization at the full matrix level.
I just show such a simple system as way of demonstrating a number of concepts.
Surprisingly though, back-propagation seems to tolerate that kind of coarse parameter switching quite well.
It is a fully supervised system where back-propagation would seem to be capable of doing at least basic factorization. For unsupervised learning, well I don’t know.
The other key point is that we are continuously building and changing structure between neurons that enable the brain to use such low power and high speed as we simply follow that structure once built, but the structure is constantly evolving as we learn. This is entirely different than the GPT approach where there is a structure and what changes are “weights” on the edges etc with massive statistics - this is not at all how the brain works.
There is an extreme mechanistic position on AI that has not properly been explored.
1/ I think people have not realized you can chain linear mappings provided some other mechanism can mutate or choose the exact linear mapping (matrix,linear associative memory) at each position in the chain in a data dependent way.
2/ It is also non-obvious that backpropagation would still work very well under such circumstances.
3/ The historical understanding of neural networks fails to see the switching and gating effects of activation functions. ReLU can be view as gating, saturating activation functions in saturation become invariant to small changes in the input and act as gating. The question is gating what, selecting what? And the answer is selecting which parameters to use in a linear mapping.
There was this speculation:https://archive.org/details/atlas-lsh
The idea is the raw input is converted into coarse context and that context tells parts of the brain to become active (a routing plan.)
Then those regions of the brain allow the fine data to flow through like with the linear mapping compositions.
Training then is supposed to happen with some process similar to Direct Feedback Alignment.
Just pure speculation but somewhat biologically plausible.
@iamr0b0tx: You’re spot on that information retrieval is at the core. But rather than just working as a lookup table, it acts as the computational engine for operating on SDRs.
You can store and retrieve simple hetero-associations x → y, but also auto-associations of bundled tokens, such as (a b c) → (a b c) and (c x y) → (c x y). The combination of these edges essentially forms a hypergraph.
Set theory predicts six basic forms of auto-associations, some with generative behavior. For example, retrieval with an input (a b) yields (c), and a subsequent query with (c) yields either (a b) or (x y) — introducing a stochastic element to the retrieval method.
Sequential querying naturally traverses the hypergraph. Here is a simple experiment from the project that illustrates this behavior: Creating Intelligence
Do you think this hypergraph traversal mechanism can be a step towards multi-hop reasoning?
Do you think this hypergraph traversal mechanism can be a step towards multi-hop reasoning?
There is indeed a case where it could work for multi hop reasoning if you frame it as a graph transversal problem but I think you would need to do a lot of work to set up the graph correctly to make it work for this. This may involve making knowledge graphs and doing some GOFAI work
It was intriguing to see what you did there, reminds me of diffusion but I think one of the core mechanisms we need for AGI to even work is unsupervised learning. We need a system that can construct and maintain its own graph
What I mean by this is we need a system that can learn intermediate representations that turn out to be useful. I like to think of it like taking a road that turns out to be one that useful not just for solving task at hand but other tasks you come across later.
The “road” here would be an intermediate transformation between x and y that aids generalization. I think this is fundamentally what learning. Finding and building upon these intermediate step is also part of learning. This is equivalent to what backprop does with weights and layers but its not good enough because its all offline and trying to fit a dataset that it assumes to be representative of all possible situations it would ever be in.
I don’t see any evidence in the approach you propose that will do this kind of learning, One would have to assume that it will stumble on the intermediate transformation by doing the walk you suggested but I am not convinced it will. For it to do that it would mean the space of input representation is in the proximity of the space of intermediate transformation but I don’t see any guarantee of that. Another way I imagine it is like making a neural network of random layers and assuming that stacking enough of the layers would approximate the function you are trying to learn.
Disclaimer: I love you idea, I hope my points don’t come across as harsh. I spent several months considering VSA/SDR/Hyper vectors as an approach to AGI but I had to move on from it. Would love to connect and talk more about it if you like
This framework certainly supports online unsupervised learning. You can configure a memory component to always learn, or to learn only when it encounters new information (which requires a test query).
This way, an auto-associative memory can learn hypergraphs, bindings between concepts, or semantic relations on the fly.
For hetero-associative memory, there are two distinct modes of unsupervised learning: Heteroencoding (the “pattern separation” mechanism), and prediction (temporal memory similar to HTM).