@marty1885 @Bitking @SeanOConnor
I think the relationship of my question to HTM is that roughly speaking, generalization in HTM, at least for an SP is equal to its memory capacity. The memory capacity is the ability of its instantaneous state/configuration/linear combination (or whatever you would like to call it) to recall inputs. Recall here means a mapping process of inputs to outputs without forgetting or kicking out memory items (input representations). Even though most probably an HTM ML engineer’s goal is to “generalize” inputs as much as it can as any ML engineer would do (e.g. in ANN’s case), the HTM models themselves seemingly are not designed for this ambitious task. For example, based on people’s testimonials in their HTM works, the HTM can only do simple ML tasks.
Now, this is the beauty of HTM models I believe is because they are very simple and are potential good candidates for intelligent and distributed computational units for a massively distributed environment. In my experience, these models don’t quite encourage engineers to intentionally overparameterize these models so to increase the generalization output because they fail quickly but can recover fast, see TM it catches up quickly with new patterns. Below is an oversimplified application that is possible due to HTM’s simplicity, and this would probably be impractical for a deep learning model.
Imagine if there is a parent SP doing online learning, and when it “reaches its capacity” it stops learning (can be a variable) but spawns a child SP and this child SP does learning and spawns another child and so on (let’s call them nodes). This simple application would ensemble healthy or primed SP’s (specialized for a set of correlated inputs) at the same time minimizing the forgetting of old memories while creating new ones. If one allows these nodes to participate in a dynamic election algorithm, in theory, they may produce good results. If one thinks about this in terms of set theory, these nodes are actually subsets, while the combination of them is the intersecting solution sets in a solution space. I could write more variations of this simple application actually.