# Triadic Memory — A Fundamental Algorithm for Cognitive Computing

**URL:** <https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763>\
**Category:** Related Papers\
**Tags:** sparse-distributed-m\
**Created:** [May 27, 2022, 9:35am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763 "2022-05-27T09:35:23Z")\
**Posts on this page:** 20\
**Page:** 7

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [July 25, 2022, 1:10pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/122 "2022-07-25T13:10:17Z")

</div>

I wish my brain wouldn’t have that limitation.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [July 25, 2022, 1:12pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/123 "2022-07-25T13:12:09Z")

</div>

Our brains are much less eager to remember everything. I mean if replicating brain’s limitations is such an important criteria, why bother remembering 30000 token long sequences?

---

<div class="post-metadata">

**Author:** ![Casey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/casey/32/2874_2.png) [@Casey](https://discourse.numenta.org/u/Casey)\
**Post date:** [July 25, 2022, 7:40pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/124 "2022-07-25T19:40:03Z")

</div>

Maybe the brain is capable of remembering everything, but doesn’t to reduce calorie usage (which is important for how well the brain works, not just starvation). Some people can remember things much better than most people (e.g. hyperthymesia). That’d be hippocampus rather than neocortex probably.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [July 25, 2022, 7:56pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/125 "2022-07-25T19:56:25Z")

</div>

Just one example: The piano part of Rachmaninov’s concerto no 3 is a 30000 token long sequence.

> **[How Does a Pianist Remember the 30,000 Notes of the 'Rach 3'?](https://www.huffpost.com/entry/piano-memory-theory_b_7610566)**
>
> Concert pianists can perform a 45 minute piece with 30,000 individual notes, that have to be performed in an absolutely particular order, with rhythmical and dynamic variability, passionately creating an emotional and formal narrative, from memory,...

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [July 25, 2022, 7:57pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/126 "2022-07-25T19:57:55Z")

</div>

Ok hyper … 61 known cases, fair enough.  
If that is relevant why not apply the same judgment to Diadic/Triadic memories themselves? Shouldn’t they get rid of “write” and “read” operations and replace them with a single function that both makes a query and a write? How the temporary memory would be possible with such a “simplification”?

---

<div class="post-metadata">

**Author:** ![david.pfx](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@david.pfx](https://discourse.numenta.org/u/david.pfx)\
**Post date:** [July 26, 2022, 3:41am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/127 "2022-07-26T03:41:19Z")

</div>

I think this goes to my question. So how precisely does this work?

- The state derived from a previously recognised sequence is stored, but where and in what form?
- That state can be queried by an ordered sequence of 3 tokens, how?
- The stored triples contain random values, do they not? How does that affect lookup?

My impression is that HTM does not have this concept of a separate isolated store of previously recognised sequences, but rather each element learns its own small subsequence. It can also recognise more than one such subsequence, and thus take part in recognising multiple longer sequences.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [July 26, 2022, 9:37am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/128 "2022-07-26T09:37:48Z")

</div>

> [@david.pfx](#):
>
> I think this goes to my question. So how precisely does this work?
> 
> - The state derived from a previously recognised sequence is stored, but where and in what form?

The key to understanding the sequence memory is the feedback loop in M1:

 ![M1](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/9/9793fac0a81af3a3e64a4cc2bea843db6688ec70.jpeg)

At each time step, a new SDR comes into the top position of this triadic memory. It’s previous value propagates down to the center position. Now these two consecutive SDRs are used to query the memory for the third (bottom) position. At this point the memory learns in a self-supervised way: If the query input has not been seen before, it fills the the bottom position with a new random SDR, otherwise with the recalled value. (The neural equivalent to deploying a new random pattern at once is to increment a “counter” on a whole dendritic segment, rather than slow synaptic learning.) The SDR at the bottom position of M1 encodes memory-internal context, it’s never a value entered from outside. The context is fed back to the center position, recursively building up sequence-specific context in the subsequent steps.

> [@](#):
>
> - That state can be queried by an ordered sequence of 3 tokens, how?

You always feed one token after the other into the circuit. In the most basic implementation with just two triadic memories, two tokens are sufficient to recall context and make a prediction. The version with three triadic memories has an additional stage that combines consecutive tokens into bigrams – this algorithm needs to see three tokens to start making predictions.

> [@](#):
>
> - The stored triples contain random values, do they not? How does that affect lookup?

When working with symbols ABCD it’s common to encode them with fixed random SDRs which are then nearly (not perfectly) orthogonal. This means there will be small overlaps, which can indeed have a rare effect on sequence prediction, in particular in ambiguous situations.

> [@](#):
>
> My impression is that HTM does not have this concept of a separate isolated store of previously recognised sequences, but rather each element learns its own small subsequence. It can also recognise more than one such subsequence, and thus take part in recognising multiple longer sequences.

The HTM temporal memory (please correct me if I’m wrong) is a component with one SDR input and one SDR output. From an electrical engineering point of view it’s clear that you cannot build interesting circuits from such a component – this requires components with three ports, such as logical gates, transistors, or … triadic memories.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [July 26, 2022, 3:33pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/129 "2022-07-26T15:33:16Z")

</div>

> [@POv](#):
>
> The HTM temporal memory (please correct me if I’m wrong) is a component with one SDR input and one SDR output. From an electrical engineering point of view it’s clear that you cannot build interesting circuits from such a component – this requires components with three ports, such as logical gates, transistors, or … triadic memories.

Actually if the input is doubled in size one half can be used as a controlling context for the other half.

In general any subset of input space bits can be regarded (or used) as a “controlling gate” for the reminder input bitspace.

The main limitation being input and output spaces are same size.

Which is no longer the case with brainblocks .  
There is an explicit [context learner](https://discourse.numenta.org/t/releasing-brainblocks-0-7-1-building-ml-applications-with-htm-like-algorithms/9926/10) class that could be a powerful twist on HTM principles.

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [July 27, 2022, 2:28pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/130 "2022-07-27T14:28:08Z")

</div>

today I use HTM RDSE for testing TemporalMemory (N=1024, P=21) for one-step prediction (having 2 TriadicMemory modules) with different sinus data.  
I put the sparse SDR, the output of the RDSE encoder, into TM.

TM learns very quickly different kind of input data and immediately predicts data if it has seen it before.

 ![Screenshot from 2022-07-27 16-14-29](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/f/fcce4d77b5b6a2f01c27184876cf3caac82e1164.png)  
 ![Screenshot from 2022-07-27 16-13-13](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/2/20f277aa063476662524461bd77111d810567fd2.png)

Thanks @POv for sharing your codes and interesting algorithms.  
I will invest more for multiple steps prediction next time.

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [August 1, 2022, 10:49am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/131 "2022-08-01T10:49:29Z")

</div>

Adding a new algorithm to the collection:

Monadic Memory is an auto-associative sparse distributed memory to be used as a  
cleanup memory and for clustering/pooling a stream of SDRs. It learns new patterns in one shot.

It’s based on two mirrored Dyadic Memory instances which effectively form a hidden layer.

The capacity is similar to the capacity of Dyadic Memory, for example 500k random patterns at n=1000 and p=10.

The algorithm works at a fixed sparsity. A sparsity-reducing version can be derived from this algorithm (more on that later).

C code is available on GitHub. PDF with examples linked below.

> <https://github.com/PeterOvermann/TriadicMemory/blob/main/Mathematica/Notebooks/Monadic%20Memory%20-%20Getting%20Started.pdf>

---

<div class="post-metadata">

**Author:** ![Casey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/casey/32/2874_2.png) [@Casey](https://discourse.numenta.org/u/Casey)\
**Post date:** [August 2, 2022, 7:14am UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/132 "2022-08-02T07:14:29Z")

</div>

> [@cezar\_t](#):
>
> Ok hyper … 61 known cases, fair enough.  
> If that is relevant why not apply the same judgment to Diadic/Triadic memories themselves?

Triadic memory doesn’t always learn, but that’s just when it already knows it perfectly. It doesn’t treat the digits of pi as a distinct episodic memory. Is that what you meant? I guess if it isn’t like episodic memory, maybe it shouldn’t be doing one-shot learning.

> [@POv](#):
>
> My question to the neuroscientists here: If you rotate those diagrams by 90 degrees, will the dataflow between horizontal layers resemble what’s going on in the neocortex?

(I’m not a neuroscientist and this is just brainstorming based on memory.)

It’s hard to compare the triadic temporal memory circuit with biology. It’s ambiguous because many parts of it have corresponding SDRs. Those could be the same neurons at different points in time.

The delay mechanism is interesting. Before updating the sequence-so-far, it associates it with the current (already updated) input. When it updates the sequence-so-far cells, those cells implicate the next input.

In biology, it’d probably update the sequence-so-far near the end of an oscillatory cycle (timestep). @LiorA talked about something related in the hippocampus (HC), I think.

In neocortex, L6ct cells could be the sequence-so-far cells. They seem to suppress a surprise signal in thalamus, so they might implicate the next input. They also mostly target inhibitory cells in cortex, so they could trigger the end of the oscillatory cycle.

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [August 4, 2022, 12:25pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/133 "2022-08-04T12:25:09Z")

</div>

@POv Could you please share the asymmetric C-Version of Dyadic Memory? Thanks

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [August 4, 2022, 12:57pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/134 "2022-08-04T12:57:51Z")

</div>

just pushed it on GitHub

---

<div class="post-metadata">

**Author:** ![LiorA](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/liora/32/1671_2.png) [@LiorA](https://discourse.numenta.org/u/LiorA)\
**Post date:** [August 4, 2022, 2:16pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/135 "2022-08-04T14:16:52Z")

</div>

Cool!  
Continuing the HC analogy, If you make p high enough it will act like the Dentate Gyrus in the HC. It will take similar memory (comprised of similar sensorial representations) and return (or forward to CA3/CA1 & back to EC) a well separated representation of one episodic memory…  
for example, you enter a room… your eyes are sending similar & close views of the environment, yet you remember one room only, and not all the “different” rooms your eyes saw …  
I think this “module” is a needed one on top of the previous memories that remember everything (like the numbers of pi) regardless of their emotional or significance importance. People who remember everything are very rare …

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [August 4, 2022, 2:23pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/136 "2022-08-04T14:23:20Z")

</div>

@POv thanks  
Now it is unclear for me with your calculation of storage locations.

static int storagelocation(int n, int i, int j)  
{  
return n \* (-2 - 3_i - i_i + 2_j + 2_i\*n) / 2; // cell in base triangle  
}

According to your neuronal net in pp 3 Triadic Memory Docu, the index of the hidden unit, which connects to bit i and j of the input bits should be  
i \* N - summe of all integer from 0 to i  
or  
i_N - i_(i+1)/2

Am I right here?  
Thanks

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [August 4, 2022, 2:38pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/137 "2022-08-04T14:38:31Z")

</div>

The calculation of storage locations is not documented in the paper. It works like this: i and j are all ordered pairs you can form from the non-zero indices in x. In the C code, indices run from 0 to n-1. Therefore i runs from 0 to n-2, and j from i+1 to n-1. The formula in storagelocations maps this triangle to a linear storage array. (This triangle, by the way, is half of the triadic memory storage cube).

Hope this helps…

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [August 4, 2022, 2:42pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/138 "2022-08-04T14:42:59Z")

</div>

@POv thanks for your response.  
But my algo is immediately for this purpose:  
The storage location index = j + _N - i_(i+1)/2 for all j = i+1 to N

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [August 4, 2022, 3:26pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/139 "2022-08-04T15:26:13Z")

</div>

> [@thanh-binh.to](#):
>
> The storage location index = j + _N - i_(i+1)/2 for all j = i+1 to N

I think bit pair addressing logic deserves a topic on its own. Different structures - diadic, triadic memories and ID & value maps - all use similar addressing, the differences being in **what** is stored at these addresses.

In a sense each bit pair address is like a micro-dendrite with only two synapses, each looking at a different bit in the input SDR

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [August 4, 2022, 4:29pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/140 "2022-08-04T16:29:50Z")

</div>

@cezar_t @POv  
You are right that we have different ways for pairing them. I only tried to understand your algorithm. I tested your algorithm for the case N=7, but can not find all possible pairs between 7 input nodes and 21 hidden nodes

---

<div class="post-metadata">

**Author:** ![POv](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/pov/32/6709_2.png) [@POv](https://discourse.numenta.org/u/POv)\
**Post date:** [August 4, 2022, 7:41pm UTC](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763/141 "2022-08-04T19:41:06Z")

</div>

@thanh-binh.to There was actually a bug in the asymmetric version … thanks for pointing this out!

[Previous page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=6)

[Next page](https://discourse.numenta.org/t/triadic-memory-a-fundamental-algorithm-for-cognitive-computing/9763.md?page=8)