# Confusion in synapse count in TM

**URL:** <https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405>\
**Category:** Numenta Theory\
**Tags:** sequence-memory, question, synapses\
**Created:** [April 12, 2020, 8:18am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405 "2020-04-12T08:18:17Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![baymaxx](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/baymaxx/32/5383_2.png) [@baymaxx](https://discourse.numenta.org/u/baymaxx)\
**Post date:** [April 12, 2020, 8:18am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/1 "2020-04-12T08:18:17Z")

</div>

![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/d/d8f166536ffab4b650f5f146c6d6084d04ed3a91.png)  
I am looking at the pseudocode of TM in BAMI. I have this confusion in line 22 about the meaning of `numActivePotentialSynapses`. Does it return the number of both ACTIVE synapses and POTENTIAL synapses connecting to active cells in previous timestep?

If thats the case, then we cannot say that the segment will definitely have SYNAPSE\_SAMPLE\_SIZE **active** synapses after growing segments since potential synapses are even part of the SYNAPSE\_SAMPLE\_SIZE.  
My argument is that line 22 should be just `newSynapseCount = (SYNAPSE_SAMPLE_SIZE - 23. numActiveSynapses(t-1, segment))`

Please correct me if I am wrong.

---

<div class="post-metadata">

**Author:** ![hsgo](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@hsgo](https://discourse.numenta.org/u/hsgo)\
**Post date:** [April 12, 2020, 12:26pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/2 "2020-04-12T12:26:07Z")

</div>

The potential synapses would eventually learn to make a connection.  
So in the end, the number of active synapses would be approximately SYNAPSE\_SAMPLE\_SIZE.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 12, 2020, 7:35pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/3 "2020-04-12T19:35:02Z")

</div>

> [@baymaxx](#):
>
> the meaning of `numActivePotentialSynapses`

This is the number of already-occupied synapses on a segment. All segments have a maximum number of synapses they can grow at one time (SYNAPSE\_SAMPLE\_SIZE). So the number of previouslyActiveCells that a segment will connect to this time (newSynapseCount) is max - existing (or SYNAPSE\_SAMPLE\_SIZE - numActivePotentialSynapses).

---

<div class="post-metadata">

**Author:** ![baymaxx](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/baymaxx/32/5383_2.png) [@baymaxx](https://discourse.numenta.org/u/baymaxx)\
**Post date:** [April 13, 2020, 5:16am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/4 "2020-04-13T05:16:35Z")

</div>

> [@sheiser1](#):
>
> numActivePotentialSynapses

Thanks. I understood. Does the potential synapse in ActivePotentialSynapse point to only the potential synapses connecting to the active cells of the previous timestep or it can include any potential synapse? The latter seem to be the case from your reply. Just want to confirm. The term `active` confuses me here.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 13, 2020, 5:52am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/5 "2020-04-13T05:52:57Z")

</div>

Right good question. For a segment the `numActivePotentialSynapses` is the number of synapses who’s pre-synaptic cells are active at the given tilmestep (t-1) as shown in line 23.

Here’s how `numActivePotentialSynapses` is populated from the pseudocode:

 ![Screen Shot 2020-04-12 at 10.49.02 PM](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/2/2c440263d4f90ded059386c689f3f845777624bc.png)

---

<div class="post-metadata">

**Author:** ![baymaxx](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/baymaxx/32/5383_2.png) [@baymaxx](https://discourse.numenta.org/u/baymaxx)\
**Post date:** [April 13, 2020, 7:33am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/6 "2020-04-13T07:33:52Z")

</div>

> [@sheiser1](#):
>
> Here’s how `numActivePotentialSynapses` is populated from the pseudocode:

Understood. Thanks. In that case, should line 22 be

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/2/25dcd54ee8a17aa8455fae13e55399d043a00ee9.png)

newSynapseCount = (SYNAPSE\_SAMPLE\_SIZE - **numActiveConnectedSynapses** -numActivePotentialSynapses(t-1, segment))

We should even subtract the connected and potential synapses to get the number of new synapses to grow…right? The definition of SYNAPSE\_SAMPLE\_SIZE says the desired number of **active synapses**

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/8/8e4d3e89c8f6e301bcf16cde6adcccfe934d533b.png)

---

<div class="post-metadata">

**Author:** ![Paul\_Lamb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/paul_lamb/32/2725_2.png) [@Paul\_Lamb](https://discourse.numenta.org/u/Paul_Lamb)\
**Post date:** [April 13, 2020, 11:17am UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/7 "2020-04-13T11:17:23Z")

</div>

> [@baymaxx](#):
>
> newSynapseCount = (SYNAPSE\_SAMPLE\_SIZE - **numActiveConnectedSynapses** -numActivePotentialSynapses(t-1, segment))

No, see lines 63 and 64. Variable “numActivePotential” is incremented whether or not the synapse is above the connected permanence. Thus, this count already includes the active connected synapses (no need to subtract that number twice).

---

<div class="post-metadata">

**Author:** ![baymaxx](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/baymaxx/32/5383_2.png) [@baymaxx](https://discourse.numenta.org/u/baymaxx)\
**Post date:** [April 13, 2020, 3:59pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/8 "2020-04-13T15:59:37Z")

</div>

Thanks for the clarification. It helped me understand better. One final query:

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/b/b045eadcdbbb9f3e90b181dd377d6e2979ee13cf.png)  
Suppose a column (say A) bursts at time t. All the cells in the column are set as active. Assuming the algorithm is running for time (t), suppose that another column (say B) bursts in the next timestep(t+1). Then, the learning segment’s synapses in time t+1 will increase the permanence values to **all** the active cells in column A. But since the winner cell at time (t) in column A is the representation of the input, shouldnt learning segment at time t+1 just increase permanence values to the winner cell instead of all active cells in column A?  
I mean line 41 should be  
`41. if synapse.presynapticCell in winnerCell(t-1)`. Please clarify.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 13, 2020, 7:01pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/9 "2020-04-13T19:01:20Z")

</div>

> [@baymaxx](#):
>
> the learning segment’s synapses in time t+1 will increase the permanence values to **all** the active cells in column A

Good question. This shouldn’t be a problem in practice, since the learning segment won’t actually have synapses to all the cells in any column. Within the `growNewSegment` function, new synapses are only grown to `prevWinnerCells` which are a subset of `prevActiveCells`. This level of detail isn’t in this pseudocode, but you can find it here in the `_growSynapses` classmethod:

 ![Screen Shot 2020-04-13 at 11.57.22 AM](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/4/4c4eccfc5824e3cc50de624529eed05d0b404b93.png)

> <https://github.com/numenta/nupic-legacy/blob/master/src/nupic/algorithms/temporal_memory.py>

---

<div class="post-metadata">

**Author:** ![Paul\_Lamb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/paul_lamb/32/2725_2.png) [@Paul\_Lamb](https://discourse.numenta.org/u/Paul_Lamb)\
**Post date:** [April 13, 2020, 9:47pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/10 "2020-04-13T21:47:54Z")

</div>

> [@sheiser1](#):
>
> the learning segment won’t actually have synapses to all the cells in any column. Within the `growNewSegment` function, new synapses are only grown to `prevWinnerCells` which are a subset of `prevActiveCells`

Not sure if you are interested in differences between implementations @baymaxx, but this is one detail that some implementations differ on (for example [Etaler](https://github.com/etaler/Etaler)). There is an advantage of connecting with any of the previously active cells rather than only the winner cells – it allows repeating sequences to eventually stabilize with enough iterations. The drawback is that it can lead to more ambiguity, so when implementing it this way, you would probably also want to implement a decay of unused synapses over time.

---

<div class="post-metadata">

**Author:** ![ivansiiito](https://avatars.discourse-cdn.com/v4/letter/i/9fc348/32.png) [@ivansiiito](https://discourse.numenta.org/u/ivansiiito)\
**Post date:** [November 24, 2020, 2:10pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/11 "2020-11-24T14:10:41Z")

</div>

Hi!

I had the same question as @baymaxx and I ended up in this thread. But I still have another question. If `numActivePotentialSynapses` is the number of synapses (independently of their permanence value) that are connected with active cells (as can be seen in lines 58-64), then, couldn’t the number of synapses in a segment grow up to values larger than `SYNAPSE_SAMPLE_SIZE`?

I understand that `SYNAPSE_SAMPLE_SIZE` is the maximum number of synapses that a segment my have (am I wrong?), but if in lines 22-23 you only count the synapses which are connected to active cells in t-1, your `newSynapseCount` value, added to the current number of synapses in the segment, could easily be bigger than `SYNAPSE_SAMPLE_SIZE`. And in that case, `SYNAPSE_SAMPLE_SIZE` will not be the maximum number of synapses in a segment.

Another minor question is about line 63:  
`if synapse.permanence >= 0 then`

How could a permanence be less than 0? I think this line doesn’t really add anything (but a bit of confussion). Am I right?

Thanks a lot.

EDIT: I have been checking the code at [https://github.com/numenta/nupic.core/blob/master/src/nupic/algorithms/TemporalMemory.cpp](https://github.com/numenta/nupic.core/blob/master/src/nupic/algorithms/TemporalMemory.cpp) and apparently, the value of `SYNAPSE_SAMPLE_SIZE` seems to be equivalente to `maxNewSynapseCount`, which is a limit to the number of synapses that might be added to a segment in a single timestep. In the code there is another variable, `maxSynapsesPerSegment`, which seems to be the overal highest number of synapses that a segment can contain (which does not appear in the pseudocode). So, when new synapses are grown in a segment, there are several limits:

- The value of `maxSynapsesPerSegment`, which cannot be exceeded at any circumstance, and marks the highest ever number of synapses a distal dendritic segment can have.
- The value of `maxNewSynapseCount` (equivalent to the `SYNAPSE_SAMPLE_SIZE` of the pseudocode), which is the maximum number of synapses connected to active cells that a segment might have in order to proceed with the synapsis grow. If a winner cell has at the moment of synapsis grow a lower number of synapses connected to active cells, then it can proceed to the addition of new synapses so that after the synapsis grow it will have at most `SYNAPSE_SAMPLE_SIZE` synapses connected to active cells.
- As all the synapses are connected to winner cells, the number of winner cells also might limit the total number of new synapses created, especially since a segment can only have a single sinapsis to a given cell.

I would highly appreciate if somebody could check that I got it right.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [November 25, 2020, 10:58pm UTC](https://discourse.numenta.org/t/confusion-in-synapse-count-in-tm/7405/12 "2020-11-25T22:58:42Z")

</div>

> [@ivansiiito](#):
>
> In the code there is another variable, `maxSynapsesPerSegment` , which seems to be the overal highest number of synapses that a segment can contain (which does not appear in the pseudocode).

Yes correct, this value isn’t there in the pseudocode though it is involved.

> [@ivansiiito](#):
>
> So, when new synapses are grown in a segment, there are several limits:

Right on all counts yes, to the best of my knowledge.

`maxSynapsesPerSegment` → the total number of synapses a segment can ever have – so it won’t activate from an unlimited number of prior inputs.

`maxNewSynapseCount` → the highest number of new synapses that can be grown on a segment at any 1 time. Any WinnerCells(t-1) are added as new synapses on the segment – excluding any already there and capped by the `maxSynapsesPerSegment` limit.