# Triadic Memory — A Fundamental Algorithm for Cognitive Computing

**URL:** <https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763>\
**Category:** Related Papers\
**Tags:** sparse-distributed-m\
**Created:** [May 27, 2022, 9:35am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763 "2022-05-27T09:35:23Z")\
**Posts on this page:** 20\
**Page:** 12

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 12:27am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/222 "2022-09-20T00:27:58Z")

</div>

just so we can compare.

My implementation takes 60s to store a million random patterns and 250s to Z-read all of them witn N=1000 on a Ryzen3 3.7Ghz cpu and 2666Mhz RAM.

thats around 16000 writes/s and 4000 z-reads/s

I’m sure it can go faster though since the code is partially implemented in python.

what speeds are you getting?

edit: corrected values.

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 3:59am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/223 "2022-09-20T03:59:34Z")

</div>

If I just time the writes, to write one million _triples_ (so three million SDRs) takes about 40 seconds, or, about 25k writes/sec. That’s using SDRs with N=1000 and P=20.

In terms of operations/second, I’m getting, with those SDRs:

writes/sec: 25k  
x reads/sec: 1100  
y reads/sec: 2800  
z reads/sec: 4500

Each of those metrics roughly triples if I set P=10 (though writes go to 178k/sec), but I’ve found that with single bit connections in the memories, larger P values work better.

My hardware is AMD Ryzen 9 3900X 3.8 GHz. I can’t find the specs for the RAM, though. Reasonably fast DDR3 or 4 from a couple years ago?

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 4:18am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/224 "2022-09-20T04:18:24Z")

</div>

@JarvisGoBrr , if you’d like to replicate, do the following:

1. install Rust via [https://rustup.rs](https://rustup.rs)
2. clone my repo from [Joe Ardent / triadic rust · GitLab](https://gitlab.com/nebkor/triadic-rust/)
3. `cd triadic-rust`
4. run `cargo run --release --example=triadic -- -i 5000 -P 20 -N 1000 -t 20000` which will store 20k triples, reporting write and recall rates every 5k triples. When I run it, I get:

```nohighlight
cargo run --release --example=triadic -- -i 5000 -P 20 -N 1000 -t 20000
warning: unused manifest key: package.maintenance
    Finished release [optimized] target(s) in 0.02s
     Running `target/x86_64-unknown-linux-gnu/release/examples/triadic -i 5000 -P 20 -N 1000 -t 20000`
5000,1000,20,25641,1091,2837,4500,20,20,20,20,20,20,
10000,1000,20,25380,1096,2836,4504,20,20,20,20,20,20,
15000,1000,20,25641,1101,2844,4504,20,20,20,20,20,20,
20000,1000,20,25510,1099,2832,4504,20,20,20,20,20,20,

```

The columns are

```nohighlight
triples so far,N,P,writes/sec,x/sec,y/sec,z/sec,...

```

and the last six columns are maximum populations in recalled SDRs and minimum overlaps between recalled and stored.

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 5:01am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/225 "2022-09-20T05:01:08Z")

</div>

> [@nebkor](#):
>
> First, the baseline. 40 sequences, each 50 tokens long, using the same 50 tokens for each sequence, N=1000, P=20 (similar qualitative behavior exists for sparser P):

I wanted to call attention to the numbers from this comment. The best case perfect recall for this memory system, using 8-bit connection weights, was roughly 300 elements in a sequence, out of a possible 2000 positions.

My current system, using 1-bit connection weights, and no random SDRs in context creation, gets the following numbers:

```plaintext
$ cargo run --release --example=sequence -- -N 1000 -P 20 -s 40 -l 50 |awk '{print $NF}' |sort |uniq -c |sort -k 2 -rn
   Compiling triadic-memory v0.1.0 (/home/ardent/git/triadic_rust)
    Finished release [optimized] target(s) in 0.76s
     Running `target/x86_64-unknown-linux-gnu/release/examples/sequence -N 1000 -P 20 -s 40 -l 50`
   1432 20
      1 15
      1 14
      2 13
      2 12
      3 11
      3 10
      4 9
      1 7
      3 6
     10 5
     20 4
     37 3
     92 2
    119 1
    190 0

```

Note the nearly 5x improvement in perfect recall.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 20, 2022, 8:30am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/226 "2022-09-20T08:30:31Z")

</div>

> [@nebkor](#):
>
> and no random SDRs in context creation

So how does the context in your version works?

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 20, 2022, 10:07am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/227 "2022-09-20T10:07:47Z")

</div>

> [@JarvisGoBrr](#):
>
> to be honest, I believe you can, I’m just lazy, @POv is already doing it in his implementation, I don’t think replacing uint8\_t with uint64\_t will make much difference for his code.

You’re right, it makes no difference.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 20, 2022, 11:14am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/228 "2022-09-20T11:14:00Z")

</div>

> [@nebkor](#):
>
> If I just time the writes, to write one million _triples_ (so three million SDRs) takes about 40 seconds, or, about 25k writes/sec. That’s using SDRs with N=1000 and P=20.
> 
> In terms of operations/second, I’m getting, with those SDRs:
> 
> writes/sec: 25k  
> x reads/sec: 1100  
> y reads/sec: 2800  
> z reads/sec: 4500
> 
> Each of those metrics roughly triples if I set P=10 (though writes go to 178k/sec), but I’ve found that with single bit connections in the memories, larger P values work better.

Your Rust benchmarks look impressive.

One caveat: At N=1000 and P=20, one million random triples exceeds the capacity. I’d stick to N=1000 and P=10 for performance comparisons.

On an Apple M1 system with the 1-bit C implementation:

writes/sec: 117k  
x reads/sec: 4.3k  
y reads/sec: 12k  
z reads/sec: 14.5k

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 12:17pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/229 "2022-09-20T12:17:49Z")

</div>

> [@nebkor](#):
>
> My hardware is AMD Ryzen 9 3900X 3.8 GHz. I can’t find the specs for the RAM, though. Reasonably fast DDR3 or 4 from a couple years ago?

if I remember correctly, Ryzen 9 is almost the same clockspeed but has 16x more cache than a Ryzen 3, maybe thats where the difference comes from.

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 3:48pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/230 "2022-09-20T15:48:06Z")

</div>

Using the previous input as the new context: [Use previous input as default new context. (36549793) · Commits · Joe Ardent / triadic rust · GitLab](https://gitlab.com/nebkor/triadic-rust/-/commit/36549793839bb82a4f37854537cfc4d8b9055586#8e1c788e6167556cb86819009fec108cfe023cd5_72_65)

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 3:51pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/231 "2022-09-20T15:51:43Z")

</div>

> [@POv](#):
>
> On an Apple M1 system with the 1-bit C implementation:
> 
> writes/sec: 117k  
> x reads/sec: 4.3k  
> y reads/sec: 12k  
> z reads/sec: 14.5k

On my system, with P=10 and N=1000, I get

writes: 178k  
x reads: 3.1k  
y reads: 8.3k  
z reads: 17k

I wonder why our Y speeds are so different 🙂

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 20, 2022, 3:54pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/232 "2022-09-20T15:54:54Z")

</div>

> [@nebkor](#):
>
> Using the previous input as the new context

Hmm, not really sure but that simplification might fail distinguishing between similar, but not identical sequences.

---

<div class="post-metadata">

**Author:** ![nebkor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/nebkor/32/6105_2.png) [@nebkor](https://discourse.numenta.org/u/nebkor)\
**Post date:** [September 20, 2022, 4:14pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/233 "2022-09-20T16:14:43Z")

</div>

I’ll test that later today. My motivation was trying to come up with something that could be used in a recall method that didn’t modify the memory stores, and this bumped the recognition perf up by nearly 50%.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 20, 2022, 4:19pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/234 "2022-09-20T16:19:22Z")

</div>

well, motivation is good 🙂

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 20, 2022, 4:24pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/235 "2022-09-20T16:24:12Z")

</div>

> [@nebkor](#):
>
> I wonder why our Y speeds are so different 🙂

The performance of y reads will be closer to x or z reads depending on 1. the system’s cache, and 2. on the inner loop’s stride length n/8.

Due to the switch from 8-bit to 1-bit storage the stride length decreased 8 times, which on some systems will be small enough to benefit from better caching behavior.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 4:42pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/236 "2022-09-20T16:42:01Z")

</div>

> [@nebkor](#):
>
> @JarvisGoBrr , if you’d like to replicate, do the following:
> 
> 1. install Rust via [https://rustup.rs](https://rustup.rs)
> 2. clone my repo from [Joe Ardent / triadic rust · GitLab](https://gitlab.com/nebkor/triadic-rust/)

I tried installing rust but I’m having some issues, my distro is probably broken or maybe I’m just dumb.

but I managed to run @POv’s C implementation and got:

- 77k writes/s
- 8.1k z reads/s

after some optimization to my code I got:

- 63k writes/s
- 6.8k z reads/s when 1M patterns are stored
- 11k z reads/s when 100k patterns are stored
- 15k z reads/s when 10k patterns are stored

I wish I had a more powerful CPU to run those tests on.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 20, 2022, 7:14pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/237 "2022-09-20T19:14:21Z")

</div>

> [@JarvisGoBrr](#):
>
> - 6.8k z reads/s when 1M patterns are stored
> - 11k z reads/s when 100k patterns are stored
> - 15k z reads/s when 10k patterns are stored
> 
> I wish I had a more powerful CPU to run those tests on.

Read performance shouldn’t depend much on the number of stored patterns.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 20, 2022, 7:16pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/238 "2022-09-20T19:16:24Z")

</div>

Depends on what optimizations it does, e.g. if a whole byte is 0 there-s no need to cut through it to see which bits are 1.

Then obviously at low population a lots of times it can skip bit finding.

---

<div class="post-metadata">

**Author:** ![JarvisGoBrr](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jarvisgobrr/32/6147_2.png) [@JarvisGoBrr](https://discourse.numenta.org/u/JarvisGoBrr)\
**Post date:** [September 20, 2022, 7:30pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/239 "2022-09-20T19:30:03Z")

</div>

> [@POv](#):
>
> Read performance shouldn’t depend much on the number of stored patterns.

My implementation uses the bit-skipping trick to ignore zeros on 64bit words I mentioned earlier.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [September 20, 2022, 7:58pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/240 "2022-09-20T19:58:03Z")

</div>

What I find a bit frustrating is how come the triadic memory handles 1M entries without error while the sequential memory on top of it has a significant failure rate at 2000

I don’t think simple increase of P from 10 to 20 should have such a high impact.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [September 20, 2022, 8:16pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/241 "2022-09-20T20:16:29Z")

</div>

The capacity of a temporal/sequence memory mostly depends on the dictionary size and the depth of the feedback lines.

If you wanted to store a sequence of 1M all-different items, the basic temporal/sequence memory algorithm would handle that without problem.

[Previous page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=11)

[Next page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=13)