# Alternative sequence memory

**URL:** <https://discourse.numenta.org/t/alternative-sequence-memory/3898>\
**Category:** Tangential Theories\
**Tags:** sequence-memory, temporal-memory\
**Created:** [May 13, 2018, 8:30pm UTC](https://discourse.numenta.org/t/alternative-sequence-memory/3898 "2018-05-13T20:30:51Z")\
**Posts on this page:** 1\
**Showing post:** 24

<div class="post-metadata">

**Author:** ![sebjwallace](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sebjwallace/32/6411_2.png) [@sebjwallace](https://discourse.numenta.org/u/sebjwallace)\
**Post date:** [May 28, 2018, 6:49pm UTC](https://discourse.numenta.org/t/alternative-sequence-memory/3898/24 "2018-05-28T18:49:44Z")

</div>

> [@Paul\_Lamb](#):
>
> > Devils advocate again - from this union I don’t know what came first. Was is it VFS, AB, CD? Or what came second or last?
> 
> Correct, from this union you cannot tell them apart. That is one reason I mentioned that union isn’t the best TP strategy (also because it quickly becomes saturated, leading to false positives)

I’ve been thinking about this TP issue and thought I’d try the other TP method we briefly discussed in another thread.

> [@Is HTM multi-step prediction currently possible?](https://discourse.numenta.org/t/is-htm-multi-step-prediction-currently-possible/3929/1):
>
> Something like an SDR is layer 3 remains stable because it is associated with all the TM SDRs of a sequence in layer 4? Then that stable representation is projected back to layer 4 to make the multi-step prediction.

> [@Is HTM multi-step prediction currently possible?](https://discourse.numenta.org/t/is-htm-multi-step-prediction-currently-possible/3929/2):
>
> If you think of a sequence as an object, apical feedback of that object represented in a pooling layer cause all the elements of the sequence (or section of the sequence, depending on the pooling implementation) to become predictive.

**The code I use for the following examples can be found in this [gist](https://gist.github.com/sebjwallace/9ebe3ae7f484dfda199a51fdf17946c7).**

**To see it in action check out this [codepen](https://codepen.io/anon/pen/yEBqVy?editors=0012)**

In the above gist I’ve written a network that does first-order-memory (layer 4) and temporal pooling (layer 3). I have used two sequences for demonstration - ABCD & XBCY. I have not implemented learning as I just want to demonstrate recall, so the weights are hard-coded. The symbols (XABCDY) are represented as single cells instead of SDRs for simplicity.

Layer 4 is simple as it just does first-order-memory. So A or X activates B, B activates C, C activates D and Y. All the weights in the matrix are binary, so the pre-synaptic neuron will activate the post-synaptic neuron instead of just depolarise/predict, I’ll explain why later. All the synapses between all the neurons in layer 3 are in the distal weight matrix.

Layer 3 has two cells to represent the two sequences. I have labelled them N & K. N represents ABCD, and K represents XBCY. While either sequence from layer 4 is playing out, the associated cell in layer 3 remains active. The synapses between layer 4 and 3 are in the proximal weight matrix. The weights are continuous. They simply represent the order in which the inputs arrive. For example, for N: A=4, B=3, C=2, D=1; for K: X:4, B=3, C=2, D=1. This simple encoding allows for competition between N & K during recall. If A was fed in from layer 4 then N will have a value of 4, and K will have a value of 0. Feed in B from layer 4, and feed back in the output from layer 3, then N will equal 43 and K will equal 3. The value for N is higher than K, and will remain that way for the duration of the sequence if C and D follow. _I’ll explain later how layer 3 cells have their activation calculated._

The output from layer 3 is fed back into layer 4 through via the apical weights. By summing the inputs and weights from the distal and apical synapses the cell with the highest value is selected to be active. This again is based on the idea of competition. After C has been fed in, N will equal 432 and K will equal 32. C activates D & Y, but as D has apical synapses from N, D will have a value equal to N, while Y will have a value equal to K, so therefore D wins out.

The proximal synapses in layer 3 detect sequences in a specific order. A cell that represents a particular sequence from layer 4 will increase in value as the sequence plays out. For ABCD, N will increment like so: N=4, N=43, N=432, N=4321. If the sequence from layer 4 is incorrect the layer 3 cell will still increment, but just not as much. So for ACBD, N will increment like so: N=4, N=42, N=423, N=4231. For DCBA, N will increment: 1, 12, 123, 1234. The value of a cell in layer 3 is calculated: _layer3[i] = layer3[i] \* 10 + layer4[j]_. So cells in layer 3 that have similar representations will have closer values than those with dissimilar representations.

In the code above I just input the beginning of the sequence (A or X) and let the memory unfold automatically over time. The in interesting thing is that if I input X, the layer 3 cell K has already predicted the whole sequence. The same for input A.

![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/0/01eafda95ce72455531906323904ac0d1910c272.png)

---

_[View the full topic](https://discourse.numenta.org/t/alternative-sequence-memory/3898)._