Hello! Im implemented spatial pooler and wish to implement temporal pooler, but it more complex and im stuck. I have a lot of questions, but at first I would like to understand role of cells in columns.
Why i cant add distal dendrit segments direct to column? Why im need add dendrit segments to cells?
Im understand that cells represent different contexts, but why each distal dendrit segment cant represent context?
At this moment i think that cells just… unnecessary abstraction.
I don’t think its a dumb question. It’s true that both cells and distal segments are used to distinguish known input categories by sequential context. So I don’t see why both structures are strictly needed, from an ML practitioner’s perspective at least. You could just have a 1-cell column with many segments, and it could function the same – as long as the segment synapses connect to other segments (not just to columns).
Biologically its known that mini-columns have a certain number of cells, and since Numenta does “biologically-contained” learning it wouldn’t ignore this whole structure. This is my understanding of Numenta’s stance, and I’m definitely not speaking for them.
I for one would be curious to see this 1-cell column version compared to the common version – to see if we’re right about equivalent performance.
This is what Numenta calls Single order (one cell per minicolumn) vs High order (many cells per minicolumn) memory. Take a look at this episode of HTM School:
Each distal segment represents context, but the cell that contains them will activate for all of these contexts. As the TM works, the prediction is achieved through the activity of the cells in the previous instant. So you won’t get a high order memory with just one cell per column. What Sheiser proposed, in a non-biological approach, was to achieve prediction through the activity of distal segments and not cells, and thus achieve high order memory without depending on the cells.
I think there’s a preference for completely separating the representation of an external sensory input (i.e. minicolumn activity) from the representation of the internal temporal context (i.e. cells). Giving minicolumns distal segments is an interesting idea! I believe it’s not done because that would imply temporal sensitivity in the spatial pooler, and mixing spatial with temporal data in the same structure makes things harder. Just my two cents!
Not to speak for OP, but I believe they were saying that in a situation where multiple contexts are not needed, having the cells at all is unnecessary.
This sounds like an interesting idea. I’m not sure I completely grasp it yet, though. I’m having trouble visualizing how predictions would work in this setup. If distal segments are outputting to other distal segments, that seems like transmitting activity (versus vanilla HTM in which predictions cannot be transmitted). I’m guessing the solution has something to do with the “1-cell column”, but having some trouble visualizing it.
With vanilla TM distal segments become active, making their cells predictive. I guess I don’t see why the distal segments themselves couldn’t become predictive. So if a distal segment is predictive and then its column is activated, that segment could activate and inhibit all other segments on the column – as would normally happen with cells.
The premise is that we don’t strictly need 2 structures to encode sequential context (cells and segments). For its core functionality, HTM separates inputs’ spatial traits through activating different sets of columns (S.P.) and temporal traits by activating different versions of these columns (T.M.) – allowing inputs which share spatial traits to be distinguished by context. These different versions are normally embodied by different activated cells within the columns.
My point (and @bizunow13’s too I think) is that these different versions can be captured by segments alone. It should work as long as one activation of a column can be distinguished from another. This system is of course less bio-realistic, but I think it would behave the same in ML practice. But maybe I’m missing something? I’d be curious to see this structure implemented on the HotGym data to find out.
I see. In that case, I don’t think you would need one cell per minicolumn either (cells could just go away, leaving minicolumns and segments).
One potential issue I see is that this would function similarly to vanilla HTM configured to have max segments per cell set to one. There are some other conversations on the forum related to that configuration (for example here).
Of course, one big difference between this idea and the one-segment-per-cell idea, is that this strategy would also be equivalent to your idea of growing new cells on the fly. I’ve not seen that particular implementation before, so not sure about the benefits or drawbacks.
Not quite. The role of the minicolumns in this is to have collections of cells (or collections of segments in this weird case) which share proximal connections, saving processing power. I’ve explored HTM without minicolumns, and the main disadvantage is that it costs about 32 times more processing power to do spatial pooling (assuming a typical layer configuration).
Of course you could just rename “minicolumn” to “cell” and call it a day, but you would also need to change the nature of the cell (i.e. there would be no prediction on it, and it would have no axon)
There are two possible scenarios I can think of right now: First: You will flatten the 3D architecture of the HTM region, where a cell (or a column) will be the core computational unit. Each cell will be equipped with its own set of dendritic trees (proximal and distal) and will act independently on the cellular activities that occur within or outside the region.
Pros: Probably there are some potential advantages but I can not think of one right now!
Cons:
1- Boosting at the cell level rather than column level
2- More computations. The HTM model will be much slower.
3- You need to define local inhibition regions because there is a possibility that you will lose the feature of distribution in the representation, i.e. the activities of the cells can be clustered in small regions rather than scattered all over the place.
Second: You merge all the cells encapsulated by a single column into one cell, in which segments (particularly distal segments) can be in an active and predictive state.
Pros: same as above!
Cons:
1- You will end up with weak neuronal representations. Do not forget that the processing does not end at the HTM region. There must be a classifier to translate the neuronal activities into a conventional data format (e.g. SDR classifier).
2- Most likely, you will also need more complicated classifier and learning rule to recognize the SDR combinations generated by HTM regions (No more overlap in the representations i.e. in following sequences ABC and XBZ, B in these two sequences can have totally different neuronal representations)
I’m not seeing such a drastic change in the algorithm as a basic requirement. Essentially all @sheiser1 is talking about is merging the concept of distal segment and cell into one abstraction just called “segment”. Segments would have distal synapses as input, proximal input, and an “axon” output. They could be in a predictive or active state.
By eliminating the concept of cells, and migrating all of their features over to the segments, one could implement HTM in a couple of ways that would probably not be a significant deviation from vanilla HTM:
Keep the concept of minicolumns around, and create all of the segments ahead of time (for example 32 per minicolumn). The minicolumns would manage the proximal input for all of their constituent segments. The algorithm in this case would behave exactly the same as vanilla HTM configured with max segments per cell = 1.
Keep the concept of minicolumns around, but only create segments as needed. The minicolumns would still manage the proximal input for all of their constituent segments. This algorithm would behave as the alternate implementation suggested by @sheiser1 in his earlier thread, where new cells are created on the fly, and they have one distal segment each.
