There are no continuous weights in this model.
The associative memory for sparse representations lies in the network topology,
assuming binary neural connnections.
So this is a complete departure from neural networks: no need for matrix multiplication,
backprop, incremental learning, or GPUs.
Ok, to initialize the topology. I don’t think quantization is a big difference, they use 1bit weights in ANNs, that’s the same as your Booleans?
This architecture is fundamentally different than ANNs, even when quantized to a single bit.
@POv Good work!
This dove-tails very nicely with my Discrete Cortical Circuits (DCCs) framework. Many parts of the computation and representation can be viewed as finite set operations. I have been missing a big chunk of the set-based mathematics in my explanations.
You can see a great deal of my published demos and media. I’ve been mostly focused on frameworks and visuals. You can find a lot of in-browser interactive demos and simulations.
I can see your work is heavily influenced by the HDVSA literature. The down-side to this is that a lot of that work is centered around data retrieval task, which is not necessarily something that you should expect in a cortical circuit. The SHR is an useful abstraction for describing a robust discrete symbol but I’m not certain its a good circuit design or biologically plausible. Many neurons performing the exact same task exclusive to any other task is not really in the spirit of things.
Anyway, i’m still reading. Thanks for sharing.
I understand that it’s local. In fact your hidden layer is a higher-order local than Hebb: lateral coincidence detection before vertical coincidence detection?
That’s actually similar to self-attention, but node-contained and binary?
And that’s why it picked my interest, this is similar to my scheme too: I have lateral cross-comp among elements of a higher node, then connectivity clustering, then centroid clustering via local feedback (Hebb is a form of centroid clustering, just online and without mandatory final convergence).
But my scheme is far more complex, with incremental orders of quantization, encoded with nested derivatives and composition levels, etc. It’s not neuromorphic, neurons are fundamentally crippled in what they can do internally. You add one hidden layer, I pack a whole indefinitely nested semantic hierarchy inside a node 
Hi Peter, glad to see you have the resources to keep going, quite of work you’ve done!
I did some experiments on bit-pair addressing algorithms on my own, with a different direction. More towards “standard” ML paradigms - e.g. regressors.
A a univariate regressor maps an P/N SDR to a single float value - it is very much like a linear regressor array with N(N-1)/2 “weights”. So instead of storing/reconstructing bit vectors at every P(P-1)/2 memory slots, it nudges/averages P(P-1)/2 float values associated with each SDR.
Since such a memory can have arbitrary vector of floats at each address point, such a primitive can also be used to implement a SDR input ->dense vector output associative memory. And because the output variables are independent from each other, there is no restriction on output vector size either, apart from required memory and penalty of averaging multiple floats.
This simple experiment showed an interesting result regarding capacity - I found out it that up to P < N/4 (which is much higher than P = N/100), the storage capacity does not depend on P. It is linear with N squared, more precisely the limit is slightly below (N**2)/3
For example a SDR_Regressor with N = 200, P =50 (4:1 “density”) can easily store a bit over 12.5k (random SDR to random float) associations. There doesn’t seem to be a limit on precision with appropriate epochs and learning rate tuning, and it also seems a faster, more accurate learner than an off-the-shelf MLP regressor from python’s scikit-learn library with similar amount of weights. But that is debatable since an MLP has more hyperparameters that can be optimized around a specific problem. Random associations was not what they were designed for.
Cheers!
This sounds interesting. I remember you mentioned experimenting with bit-pair expansion.
There isn’t a whole lot of research literature on combinatorially expanded topologies. I guess it seems totally counter-intuitive to square a hyperdimensional space – until you realize that this becomes manageable if activation is sparse.
Hey Mr. Overmann,
I read your paper Creating Intelligence: A Computational Foundation for AGI with great interest. Your approach of discarding continuous weight matrices in favor of discrete sets, hyperdimensional representations, and subset pattern matching closely aligns with the non-LLM cognitive architecture I am currently building.
While developing my system on discrete state models, I encountered several systemic challenges that your core algorithm might also face as it scales. I wanted to share a few architectural mechanisms I developed that could directly complement your framework:
1. Neuromodulated Dynamic Thresholding: Subset pattern matching often struggles with fixed matching thresholds, which risk either missed associations or combinatorial explosion. I address this using simulated neuromodulation (for example, a GABA-inspired inhibition signal that dynamically raises matching thresholds during high system activity to strictly maintain sparsity).
2. Stall Detection via Lineage Observers: Pure associative matching engines risk falling into topological traps, oscillations, or state saturation. My system uses a detached “Longitudinal Core State Lineage Observer” to monitor the entropy of state transitions over time. When a stall is detected, it triggers targeted reset signals (such as noradrenaline-inspired shifts) to break out of the dead end.
3. Sleep-Replay Consolidation: Continuous path creation eventually causes graph bloat. Implementing offline sleep-replay cycles allows the system to prune low-value connections and reorganize paths based on metaplasticity signals, keeping set-matching performance optimal over time.
4. Active Gap Management: To move beyond reactive set-matching, my system explicitly tracks internal knowledge gaps as system states and evaluates progress using closure deltas. This transforms a passive pattern memory into an active, self-directed cognitive engine.
I am currently running and validating these loops within a relational SQL schema to maintain full auditability of every state change before moving to high-dimensional representations.
Given our shared focus on discrete, non-LLM cognitive foundations, I would welcome the opportunity to connect and exchange ideas.
Best regards,
What subset matching algorithm are you referring to? LSH?
I don’t understand this. If you dynamically raise matching thresholds during high activity, you’re certainly not maintaining strict sparsity.
Not LSH. Locality-Sensitive Hashing (LSH) is a probabilistic, approximate algorithm designed for nearest-neighbor vector search.
What I am referring to is exact, set-theoretic subset matching operating on discrete representations and Sparse Distributed Representations (SDRs).
In this context, subset matching relies on exact set intersections, relational joins, or bitwise overlap operations to determine whether an input feature set Q overlaps sufficiently with a candidate stored set S (measuring whether \vert{}Q \cap S\vert{} \ge \theta).
The issue with fixed overlap thresholds (\theta) in deterministic set-matching is two-fold:
1. Threshold too high: The system misses partial matches, weak associations, and valid sub-graph inclusions.
2. Threshold too low: High input activity triggers a combinatorial explosion of overlapping sets, saturating the memory graph and causing state noise.
Rather than switching to probabilistic bucket-hashing like LSH, my architecture (RNS-AI) solves this deterministically using dynamic thresholding inspired by GABAergic inhibition. This dynamically tunes the required overlap threshold (\theta) based on current system activation, preserving exact, audit-ready relational matches while strictly maintaining system sparsity.
In this context, the matching threshold (\theta) is the minimum number of overlapping features required for a candidate target set or downstream node to fire.
Here is why raising the threshold during high activity enforces sparsity:
1. The Problem: When overall system activity or input feature density is high (a dense or noisy input state), a static or low threshold allows a flood of candidate sets to match simultaneously. If 500 different stored sets suddenly meet a low threshold, downstream activation explodes, state noise spikes, and sparsity collapses.
2. The Mechanism: When the system detects this high global activity, simulated GABAergic inhibition dynamically raises the bar (increases \theta).
3. The Result: Because the threshold is now higher, only the target sets with the absolute highest degree of exact overlap can cross it. Weak or partial matches are suppressed. Fewer nodes fire downstream, which directly preserves a strict k-sparse activation state.
To put it simply:
High activity + low threshold = dozens or hundreds of sets fire (sparsity destroyed).
High activity + raised threshold = only the top fraction of a percent of exact matches fire (sparsity maintained).
This works identically to global GABAergic inhibition and k-Winners-Take-All (kWTA) mechanisms in biology: when network volume gets too loud, inhibition raises the firing threshold across the population so that only the strongest, most precise signals get through.
Your comment’s edit reveals that this is LLM generated. Let’s not waste our time.
Yes, that may be, I never said anything else. I’m from Germany, I’m not a scientist or anything. I have dealt deeply with the subject. My English is not very good. Especially not my technical English. That’s why I use it to communicate. Sorry if that bothers you, I just wanted to help. Nevertheless, it doesn’t changes something about the content of the answer.
Also the code in your repo is completely AI-generated…
Look, what I love about HTM Forum is that there are real people here
with deep insights, unconventional ideas, and strong opinions.
Let’s hope this doesn’t change.
It would be great if you don’t confuse the use of tools with the idea.
Do you still go into the technical topic or is the meta discussion enough for you? Or do you want to end the discussion?
On the other hand Peter, consider how little a group interested in Mathematica ever discusses machine intelligence, and how much a group interested in machine intelligence ever mentions Mathematica. The two are so disjunct that any co-occurrence can be considered random overlap.
If you really want to pull interest you can leverage the darn AI to convert the github examples into the language a ML person has patience to give it a glance.
It doesn’t need to replace or override original code, just provide a different language implementation with full “AI generated section” disclaimer.
As long as instead of using AI tools to impersonate human thinking one uses them to follow and incarnate it, they can be quite useful.
The thinking is already there in text and code, AI tools are capable enough to do the conversion specially if they can test the new implementation results against the original one.
Back to your work, I don’t quite understand SHR-s
Sparse holographic representation (SHR)
A set where all possible subsets represent the same semantic information. Used as an encoding scheme for symbolic tokens.
AFAIK the empty set is a subset of all possible sets and it also cannot hold any information (semantic or not) - besides the fact that it is empty, therefore I can only assume the above definition misses something important.
Edit:
Different SHRs may randomly overlap in a few elements, but such small overlap does not indicate information similarity. Two random sparse sets are extremely unlikely to share more than a few elements.
In this case the overlap itself, regardless how small it is, breaks the definition too - its semantic information is ambiguous.
Fully agree. The code base is written with auto-conversion to Python in mind.
As one of the language’s contributors, I’m personally more productive using Mathematica. But as you say, the AI community doesn’t speak Mathematica.
To put it simply: SDRs represent perception, SHRs (basically random SDRs) represent symbols.
At the top of the cognitive hierarchy, the active elements within a sparse representation are not directly linked to low level sensor data. Instead, each active element represents the whole.
To “bind” cognitive concepts, you store bundles of SHRs in memory, creating a hypergraph.
If you remove active elements from an SHR, ambiguity arises according to the overlap probabilities. And below a certain number of elements (approximated by the “pattern matching threshold”), SHRs become indistinguishable. But we cannot define an exact activation threshold below which a representation would cease to be an SHR.
Yes, the empty set is a valid subset of an SHR. We could say that emptiness represents everything. But that’s a purely philosophical statement with no practical consequences in this framework.
A meta point: perception won’t solve cognition. I introduced SHRs as a building block for symbolic AI. The question is now, how exactly are language and knowledge represented?
HTM’s SDRs were meant as general semantic encoding. Perceptual encoding into SDRs were just a proposition on how to handle the conversion from real-life data to input layer of a machine designed to operate with SDRs. Input layer SDR encodings are emphasized because there-s no straightforward “use an autoencoder to transform any data into a vector” as in NNs, but there-s no explicit or implicit rule by which a SDR should exclusively represent input data.
So from this perspective SHR another name for a what is a SDR used in a particular context: to avoid accidental or intentional similarities between SDRs. Which might, indeed, be contrary to SDR’s core idea - to encode similarities as overlaps. If this is the case, the question is why? Normal symbols or numerical tokens already have this quality of not being similar with each other while being much cheaper to store and manipulate. What can a SHR do that a “classical” symbolic representation can’t?
SDRs are useful for perceptual encoding, but not for symbolic AI. Let’s say you encode “cat” and “dog” with two partially overlapping sparse representations. Their overlap measures similarity, but it will be too sparse to have a standalone semantic meaning on its own. You cannot extract the few overlapping bits and process them further.
Another problem is that SDR-based systems usually work with fixed population sizes – adding or removing features would break the sparsity envelope.
SHRs represent stable concepts at the top of the cognitive hierarchy. They encode symbols as random sparse representations with consistent populations.
Bundling SHRs forms a hyperedge. Think of a hyperedge as a super-SDR, where each active element corresponds to an entire SHR. Importantly, the number of SHRs within a hyperedge is variable. Using an associative item memory, you can extract the SHRs from a hyperedge and process them further.
Similar hyperedges overlap, approximately by multiples of the SHR population. The overlap between similar hyperedges again consists of symbolic SHRs, which can be isolated and further propagated.
Stored in associative memory, the combination of hyperedges results in a knowledge hypergraph. Memory retrieval effectively queries and traverses the graph. There is no analog of this for SDRs.
And because SDRs and SHRs share the same substrate (neural population codes, or sets), we can now build pipelines from perception to cognition/reasoning within the same, unified computational framework. That’s why I’m talking about AGI, and not just another flavor of machine learning.