I hypothesis that the column pooler can be modified to function as a temporal memory. I hope to test this soon. The column pooler, if you aren’t familiar, is described at:
The column pooler combines the proximal dendrites from a spatial pooler and the distal dendrites from a temporal memory, and describes how to integrate them to achieve sensory-motor inference. This post describes an extension to the column pooler which integrates the proximal & distal inputs in a different way, with the hypothesized result of forming mini-columns on an ad-hoc basis.
What is a mini-column?
- A group of cells which respond to the same feed forward input. There could be an explicit mechanism causing this, or it could be an emergent or learned property. Is there evidence that cells belong exclusively to one mini-column?
- If any cells in the group are predicted, then they fire first and suppress the unpredicted cells. This results in a variable sparsity, and mini-columns can “burst” with activity, which allows the TM to represent union of contexts.
I will model this as two rounds of competition in the column pooler, which I will call the early and late phases:
- Early phase competition is for predicted cells only.
- Late phase competition is for all cells which didn’t activate in the early phase.
The two rounds of competition have different parameters. The results of the early phase controls the parameters of late phase competition.
- Sparsity is the primary link between the two rounds:
latePhaseMaxSparsity = f ( earlyPhaseResults )
- Early phase can suppress late phase, allowing for variable sparsity and the bursting behavior.
- Late phase can be guaranteed a minimum number of activations, for the situation where the full quota of predicted cells activate, but unpredicted cells have significantly greater overlap.
- Activation thresholds are also different, and early phase effects the late phase threshold.
- If the early phase is too sparse, then lower the late phase threshold so that it will “burst” with activity.
- If the early phase is very active, raise the threshold to suppress late phase cells.
- The early phase overlaps can set the late phase activation threshold, to ensure that predicted activations suppress unpredicted cells with significantly less overlap.
- The learning rates of cells firing in the two phases can also differ and interact.
- This could allow late phase bursting cells to learn the mini-column structure. Bursting is the only time that all of the cells in a mini-column are all active at the same time, making it the ideal time to learn the feed forward input pattern.
This extension can coexist with the current notion of a column pooler. Simply set all early and late phase parameters to the same values and use the following equation for sparsity, and this extension goes away.
latePhaseMaxSparsity = maxSparsity - earlyPhaseSparsity