# Triadic Memory — A Fundamental Algorithm for Cognitive Computing

**URL:** <https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763>\
**Category:** Related Papers\
**Tags:** sparse-distributed-m\
**Created:** [May 27, 2022, 9:35am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763 "2022-05-27T09:35:23Z")\
**Posts on this page:** 20\
**Page:** 13

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 9:22pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/242 "2022-09-20T21:22:19Z")

</div>

whats the test setup for the temporal memory so I can reproduce?

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 10:17pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/243 "2022-09-20T22:17:00Z")

</div>

> [@cezar\_t](#):
>
> Hmm, not really sure but that simplification might fail distinguishing between similar, but not identical sequences.

Indeed, this performed very poorly with sequences made of SDRs that had strong overlap (my `scalar_encoder_sequence` example can construct SDRs that differ by one bit for each successive element, and using previous input as new context just went off the rails completely).

The new algorithm [uses a deterministic PRNG seeded with the latest count of how many items have been added to the current sequence](https://gitlab.com/nebkor/triadic-rust/-/commit/6094d7da81761441b2b3607f2179028a79ac88a1#8e1c788e6167556cb86819009fec108cfe023cd5_66_69), and this seems to provide a best-of-all-worlds performance quality, having no problem with long sequences of many closely related SDRs, or many short sequences from a shared, limited alphabet of tokens.

Because of the way the random context SDRs are generated, the recall method (that doesn’t modify the triadic stores) is able to generate appropriate context SDRs that are the same ones that were stored in the learning step, which helps with recall performance.

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 10:25pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/244 "2022-09-20T22:25:51Z")

</div>

What’s interesting is that my implementation, with the one-bit weights, does better with P \> 10. For example, for a single sequence of 10k random SDRs, N=1000, P=10, I get the following counts for bits of overlap, sorted by bits of overlap, descending:

```nohighlight
$ cargo run --release --example=sequence -- -N 1000 -P 10 -s 1 -l 10000 |awk '{print $NF}' |sort |uniq -c |sort -k 2 -rn
    Finished release [optimized] target(s) in 0.01s
     Running `target/x86_64-unknown-linux-gnu/release/examples/sequence -N 1000 -P 10 -s 1 -l 10000`
   6082 10
      1 9
      4 8
      5 7
      4 6
      4 5
      3 4
      5 3
     27 2
    354 1
   3509 0

```

With P=20, I get

```nohighlight
$ cargo run --release --example=sequence -- -N 1000 -P 20 -s 1 -l 10000 |awk '{print $NF}' |sort |uniq -c |sort -k 2 -rn
    Finished release [optimized] target(s) in 0.01s
     Running `target/x86_64-unknown-linux-gnu/release/examples/sequence -N 1000 -P 20 -s 1 -l 10000`
   9994 20
      1 19
      1 18
      2 13

```

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 10:26pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/245 "2022-09-20T22:26:57Z")

</div>

> [@JarvisGoBrr](#):
>
> I tried installing rust but I’m having some issues, my distro is probably broken or maybe I’m just dumb.

I doubt you’re dumb 🙂 I’m happy to help out if you’d like.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 11:25pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/246 "2022-09-20T23:25:55Z")

</div>

> [@cezar\_t](#):
>
> What I find a bit frustrating is how come the triadic memory handles 1M entries without error while the sequential memory on top of it has a significant failure rate at 2000

but I think we are loosing track of the goal here.

we are building some kind of exoteric memory storage here, but dont know how to apply it exactly.

I dont expect the human brain to remember exactly ultra long sequences of random symbols. I cant even remember my phone number.

from introspection, I think we are built to extract meaning from short snippets of sequences and infer the rules behind them and not to store 2000 sequences of random characters.

I’d like to make a memory system that can learn generate novel sequences naturally, the same way we learn to speak as if it was nothing but find it hard to memorize one page of a book.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 11:32pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/247 "2022-09-20T23:32:56Z")

</div>

> [@nebkor](#):
>
> I doubt you’re dumb 🙂 I’m happy to help out if you’d like.

I’ll lookup some tutorials later, I’ll endup trying to learn this language sooner or later, so maybe now is the time.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 2:55pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/248 "2022-09-21T14:55:17Z")

</div>

I was playing around a little and ended up implementing a version of triadic memory that works directly on symbols without sdrs or neural grids, it uses python dictionaries instead and works with tuples, strings and integers as “symbols”. so it has no overlap between symbols or capacity limits.

I think its cool to play with this in an “ideal world” so I ended up implementing a sequence memory and it works quite well.

```auto
from collections import defaultdict
from random import randint, seed
seed(0)

class IdealTriadic:
    def __init__ (self, p=1):
        self.x_data = defaultdict(set)
        self.y_data = defaultdict(set)
        self.z_data = defaultdict(set)
        self.p = p

    def addresses(self, a, b):
        for a_ in a:
            for b_ in b:
                yield a_, b_

    def write(self, x, y, z):
        for addr in self.addresses(y, z):
            self.x_data[addr].update(x)
        for addr in self.addresses(x, z):
            self.y_data[addr].update(y)
        for addr in self.addresses(x, y):
            self.z_data[addr].update(z)

    def _read(self, a, b, data):
        out = defaultdict(int)
        for addr in self.addresses(a, b):
            for o in data.get(addr, ()):
                out[o] += 1
        if not out:
            return []
        return sorted(out, key=out.get)[-self.p:]

    def read_x(self, y, z):
        return self._read(y, z, self.x_data)

    def read_y(self, x, z):
        return self._read(x, z, self.y_data)

    def read_z(self, x, y):
        return self._read(x, y, self.z_data)

class ShallowTemporalMemory:
    def __init__ (self):
        self.memory = IdealTriadic()
        self.predictive_memory = IdealTriadic()
        self.z = []

    def input(self, inp, new=True):
        self.predictive_memory.write(self.z, self.z, inp)

        x = self.z
        y = inp
        z = self.memory.read_z(x if new else x + y, y)
        if not z:
            z = [randint(0, 0xffffffffffffffff)]

        self.memory.write(x + y, y, z)
        self.z = z
        return z

    def predict(self, p=1):
        self.predictive_memory.p = p
        return self.predictive_memory.read_z(self.z, self.z)

    def reset(self):
        self.z = []

mem = ShallowTemporalMemory()

for letter in 'this is a text any text is a text and this is a text being texted by a typer this':
    inp = [letter]
    mem.input(inp)

mem.reset()
print()

for letter in 'an text and this is a text beyng texted by a typer this':
    inp = [letter]
    mem.input(inp, new=False)
    print(letter, *mem.predict())

```

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 3:03pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/249 "2022-09-21T15:03:33Z")

</div>

Yeah that is nice, I guess there are quite a bunch of attempts to directly incarnate symbolic processing.

The promise of an associative approach is it should be able to handle both “vague” representations like (encodings from) sensory inputs and symbols, together with a yet-to-be-discovered bridge between these two realms.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 3:19pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/250 "2022-09-21T15:19:29Z")

</div>

The impression I get is that Triadic memory, HTM, Spatial pooler, Assembly calculus and other variats of the same idea are just ways people have attempted to implement those symbolic processors using neural-like storage.

to be completely honest, when I look at all those, I just see differently shaped hopfield networks with enforced sparsity.

we already have several different ways to implement the memory but I think maybe it would be better to focus on the algorithm instead.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 21, 2022, 7:42pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/251 "2022-09-21T19:42:25Z")

</div>

I’d like to bring your attention back to Pentti Kanerva’s Sparse Distributed Memory, a visionary concept  
introduced more than 30 years ago.

It seems hard to find research on how one would use such a memory if there was a high-performance implementation. Did anyone say “I tried this, but SDM was too small / to slow for my purpose”?

The Dyadic Memory algorithm in its latest incarnation with 1-bit storage locations is certainly today’s  
most efficient SDM: At n=1000/p=10 one can write 250k and read 30k heteroassociations x-\>y per second. The dimension of x can be as big as 20k, while there’s no limit on the dimension of y. The usable capacity is in the millions, and the speed of read or write operations remains independent of the number of items stored.

The question is now: How can we put SDM to work in real-world applications?

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 8:59pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/252 "2022-09-21T20:59:52Z")

</div>

> [@POv](#):
>
> The question is now: How can we put SDM to work in real-world applications?

exactly, we already have some really powerful implementations for sparse vector storage and search, triadic memory is certainly the best I’ve seen so far but we have no way to actually apply it onto real-world problems.

I say that because I dont think sequence memories are the right way to approach this, we are not extracting information from the input, we are just storing the input in a new data format.

we are basically building overfitting machines so far, there has to be a better way to do it which I suspect will include some forms of lossy storage and deliberate corruption of data.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 9:01pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/253 "2022-09-21T21:01:12Z")

</div>

However Kanerva’s memory features 1000 bits long dense bit arrays, which have a representation power in the order of 1e301 possible states while 10/1000 bits sdrs have only 2.6e23 available states.

Not saying this means this is necessarily insufficient, just that it might not be as expressive as one might need.

Thinking of embedding representations in both vision and language modelling where float vectors sized between 200 and 2000 are common.

The underlying idea behind them being they need to be rich enough to encode in themselves enough information about the identity of the “thing” they represent but also its content/structure.

SDRs were intended as a simplification of the same concept, a 50/1000 sdr “floats” in a space 1e85 states, still much lower than Kanerva’s yet much higher than a 10/1000 bit representation.

```auto
from math import factorial
def sdr_states(n,p):
    return factorial(n)/factorial(n-p)/factorial(p)

print(sdr_states(1000,10))

```

PS not to say GPT-3 embeds the words (tokens) in a vector of size 12288

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 9:10pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/254 "2022-09-21T21:10:42Z")

</div>

There’s a tradeoff between how much information you need to represent vs how much hardware to do computation.

those dense representations have more content, therefore they take up more storage space, so we can fit less items in memory no matter the implementation.

I believe brains solve this problem by using clever forms of lossy data compression, somehow you need to learn whats irrelevant in the data and throw it away while storing it.

an extreme example: here’s some data, you may guess it takes a fair amount of space:  
 ![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/5/583f555aa7e133a7a7ae58c50314c069d4b411b5.png)

but I can use my brain to convert this data into a string, which has no resemblance to the original data but still better captures its meaning and takes up less space: `[120, 94, 50, 43, 121, 94, 50, 61, 49]`

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 9:45pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/255 "2022-09-21T21:45:14Z")

</div>

Yes but the question is can you represent **anything** with 9 bytes?

How many bits in a SDR are needed to encode a vector 9 int8-s, and how much more bits you need to add to the SDR for it to have some overlap with neighboring (aka similar) encodings?

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 9:50pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/256 "2022-09-21T21:50:51Z")

</div>

of course not but neither do a cortical column with 200k bytes.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 10:00pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/257 "2022-09-21T22:00:24Z")

</div>

At [Cortical.io](http://Cortical.io) they use N=16k SDRs to represent text content and is rich enough to search and retrieve similar documents with a “query” document.

Which as an example application proves that it is pretty sufficient to embed rich content and… if it were to be replicated via tri/diadic memories there are two obstacles:

- implicitly diadic or triadic associations points to a SDR, which does not tell in itself what are the actual documents we look for
- it points to a single SDR or an union of SDRs if multiple Y-s are stored with the same key X
- not to say its P size is in the hundreds, but it might get a response in a second or so.

PS by document it could be anything from an email to application forms, contracts or hundreds of pages novels, no actual restriction on content, as long it is textual

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 10:24pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/258 "2022-09-21T22:24:32Z")

</div>

thats another thing that gets me confused, triadic memory is too sparse, from sources I find online, the brain seems to work with 1-5% sparsity while triadic uses 0.01% sparsity.

maybe we are treating sdrs as symbols but the brain and [cortical.io](http://cortical.io) are working with superpositions of several symbols to deliberatelly cause interference?

not to mention, a 1000^3 block of memory would need a million neurons to be implemented, its too big for a cortical column, jeff estimates it to be 700 neurons but the biggest estimate I found is 200k neurons, which is still too few neurons. maybe we should drop the idea of a self contained column?

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 10:30pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/259 "2022-09-21T22:30:02Z")

</div>

it is 1% not 0.01% . At 100/10000 it would be pretty rich.

I would not go too far with analogies with HTM… besides the SDR representation. If we want to force that analogy I would consider “column” each bit-pair location. So an N=1000 diadic memory is an abstract representation of N\*(N-1)/2 ~ =500k micro-columns each “learning” an 1000 bit SDR

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 21, 2022, 10:36pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/260 "2022-09-21T22:36:31Z")

</div>

I was making analogies with HTM because although the algorithm is different, the limitations should be pretty similar.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 21, 2022, 10:43pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/261 "2022-09-21T22:43:43Z")

</div>

I’d wager they-re complementary.  
There-s another sense in which the bit-pair associations match thousand brains theory - their pattern recognition mechanism IS voting.

[Previous page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=12)

[Next page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=14)