@Falco and I talked about this online on Sunday, but I’ll reiterate it here for those who couldn’t make it due to other commitments.
I think I have a solid grasp on how to create a system that can create learned representations for recognized spatial patterns (features, objects, etc.). I also have a pretty decent understanding of the existing HTM model for learning temporal sequences of patterns. However, I cannot shake the feeling that I’m missing something important.
Up until recently, I’ve been focusing on using the aforementioned techniques to recognize patterns and sequences of patterns. I’ve started to realize that the more times a sensed pattern is recognized, the more it’s representation begins to settle on the average of the inputs that match to it. While this permits a very efficient way of capturing the principal components of sensed and learned patterns, it does leave open a question of what, if anything, to do with the residual information. That is the part of the sensed input signal that is distinct from (orthogonal to) the stored representations of recognized patterns.
To be clear, I’m not talking about what to do with a novel input that is not recognized by the network. An agent could easily expand its dictionary of recognized patterns when encountering a new feature or object that does not sufficiently overlap or align with any previously learned patterns. In this case, the magnitude of the residual exceeds some threshold which triggers the learning algorithm to learn and store a new pattern, of which the current residual is the first exemplar.
What’s been nagging at the back of my mind is what to do with the information contained in residuals that do not meet the threshold for learning a novel input. To my mind, this information is the part of the sensed input that makes this moment unique. It is everything about the current environment that didn’t fit neatly into existing bins. The imperfections, if you will, with respect to the idealized forms that have already been sensed, recognized and passed on for further processing in other parts of the brain.
Up to now I’ve been presuming that this residual information is simply discarded as background noise. That may very well be true; however I’ve recently started to wonder if it might be possible to use the residual to fulfill some other purpose. Potential uses could be to provide a unique signal to the lower brain structures that allows them to disambiguate two very similar moments in time. Or perhaps as a random bit of color to the HC/EC that allows it to assign a unique identity to the memory before it gets correlated and stored with other long term memories.
Alternatively, there might be some spatial information encoded in the residual. In this case, it may be that the residual is the representation that encodes a spatial heat map of the negative space (i.e. the space not occupied by recognized objects or features). This would act sort of like a disparity map, but instead of subtracting common features from a stereo pair of images, you are removing all features that have been recognized and/or are being attended to.
This last bit might help to explain the figure-ground perception problem. The features that you are attending to get filtered out for further processing, while everything else becomes background. When your change your attention, you are focusing on the parts of the input that are currently salient, while everything else goes to background. The question remains: Is there information in the background/residual that the brain is using for other purposes even if it is not the current focus of our attention?