# Sequence in temporal memory

**URL:** <https://discourse.numenta.org/t/sequence-in-temporal-memory/6868>\
**Category:** Applications\
**Created:** [December 2, 2019, 7:14am UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868 "2019-12-02T07:14:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![John\_Sorok](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/john_sorok/32/4688_2.png) [@John\_Sorok](https://discourse.numenta.org/u/John_Sorok)\
**Post date:** [December 2, 2019, 7:14am UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/1 "2019-12-02T07:14:13Z")

</div>

If we have 2048 columns and 32 cells per column, then whats is the max pattern/sequence length that TM can learn?

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [December 2, 2019, 8:34am UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/2 "2019-12-02T08:34:23Z")

</div>

**Math might be wrong, please don’t take my words for it. Please use monti-carlo cimulation**

The math is complicated. Assuming your input SDR only have 1 on bit, every column of the TM can learn 32 distinct ways for it to be on.

Consider a simpler case. We have a TM with 4 columns and 2 cells per column (and the same assumption of every SDR can only have 1 bit on). Then every column can have 2 ways for it to be on. So the sequences A-B-C-D, A-A-C-D can be learned but A-A-A-C can’t.

For a more real world cases, we assume bits in SDRs are evenly distributed with a 4% density and a sequence length of 16 (a random value I came up). Then we can calculate that on average it will take 800 different SDRs to have a bit turned on 32 times and a ~50% of any bit overlapping with another in the sequence (on average, a bit will be on 1.5 times in the sequence if it has been on). That gives us ~533 distinct sequence.

_I haven’t take the effect of activation threshold and other parameters into account. The actual number is way larger. The number I calculate is the worst case_

---

<div class="post-metadata">

**Author:** ![tahamtan.net](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/tahamtan.net/32/4690_2.png) [@tahamtan.net](https://discourse.numenta.org/u/tahamtan.net)\
**Post date:** [December 2, 2019, 10:12am UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/3 "2019-12-02T10:12:24Z")

</div>

I think at least, we have 2048^32 different sequence! but I’m not sure. I have the same question as you have.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [December 2, 2019, 1:31pm UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/4 "2019-12-02T13:31:54Z")

</div>

But the SDR has to have a certain number on in the same 40 micrometer neighborhood to be effective.  
This smoothly morphs into the next to adjacent SDR neighborhoods so this is not just the one-of-n that is suggested in the basic SDR sparse math. Topology matters.

And all of this ignores the lateral voting between cells as an AND gate. The level of the SDR has to be taken into the context of the total cell response to the incoming constellation of excitation.

---

<div class="post-metadata">

**Author:** ![Paul\_Lamb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/paul_lamb/32/2725_2.png) [@Paul\_Lamb](https://discourse.numenta.org/u/Paul_Lamb)\
**Post date:** [December 2, 2019, 2:20pm UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/5 "2019-12-02T14:20:24Z")

</div>

> [@John\_Sorok](#):
>
> f we have 2048 columns and 32 cells per column, then whats is the max pattern/sequence length that TM can learn?

How many minicolumns per input (or worded another way, what is the column sparsity?). As @marty1885 mentioned, the activation threshold is also extremely important when talking about capacity.

This matters, because you can reuse the same 32 cells in each minicolumn in different combinations per input. For example, you could create a new represenation A(33) by taking one cell from A(1), one from A(2), one from A(3) etc. to create a new unique representation for A which only overlaps the previous representations by one cell. If the activation threshold is more than one, then this overlap is too small to give you any false positives.

The capacity of a TM layer with 2048 minicolumns is highly dependent on your configuration parameters, but with the right sparsity, activation threshold, and diversity of inputs, it is astronomically high.

---

<div class="post-metadata">

**Author:** ![AMZ](https://avatars.discourse-cdn.com/v4/letter/a/65b543/32.png) [@AMZ](https://discourse.numenta.org/u/AMZ)\
**Post date:** [December 2, 2019, 4:04pm UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/6 "2019-12-02T16:04:39Z")

</div>

In the ideal scenario, HTM capacity = (n\_cols!)/(n\_wins! x (n\_cols - n\_wins)!)  
n\_cols = total number of mini-columns in the HTM region  
n\_wins = number of winning mini-columns

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [December 2, 2019, 5:55pm UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/7 "2019-12-02T17:55:35Z")

</div>

In case anyone missed it, we talked about HTM sequence capacity in the [Neuron Paper](https://www.frontiersin.org/articles/10.3389/fncir.2016.00023/full).

> [@Estimate the capacity of an HTM model](https://discourse.numenta.org/t/estimate-the-capacity-of-an-htm-model/2398):
>
> I am trying to estimate (even if it is pessimistic) the capacity of a single-layer htm model analytically. I think it depends on several characteristics of the model like the number of columns, number of cells per column and the sparsity. I have various doubts: What is the best way to express the capacity? Perhaps it can be estimated by the number of possible transitions? Is there any formula for this? Taking into account the entropy of the input data, exceeded the capacity of the model, How …

It says:

> In our model it is straightforward to obtain an estimate of sequence capacity. Although we refer to the network model as a “sequence memory,” it is actually a memory of transitions. There is no representation or concept of the length of sequences or of the number of stored sequences. The network only learns transitions between inputs. Therefore, the capacity of a network is measured by how many transitions a given network can store. This can be calculated as the product of the expected duty cycle of an individual neuron (cells per column/column sparsity) times the number of patterns each neuron can recognize on its basal dendrites.

---

<div class="post-metadata">

**Author:** ![Paul\_Lamb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/paul_lamb/32/2725_2.png) [@Paul\_Lamb](https://discourse.numenta.org/u/Paul_Lamb)\
**Post date:** [December 2, 2019, 6:18pm UTC](https://discourse.numenta.org/t/sequence-in-temporal-memory/6868/8 "2019-12-02T18:18:35Z")

</div>

> [@rhyolight](#):
>
> the product of the expected duty cycle of an individual neuron (cells per column/column sparsity) times the number of patterns each neuron can recognize on its basal dendrites.

To relate this back to my earlier post, the “number of patterns each neuron can recognize” is highly dependent (by many orders of magnitude) on a couple of configuration parameters in particular - the number of minicolumns per input (also referred to on this thread as “number of winning mini-columns” or “column sparsity”) and the activation threshold.