By @fergalbyrne
Please discuss this paper below.
Two great recent talks concerning the theory behind this paper from George Sugihara’s lab:
For HTM heads, this theory identifies what they call “singularities” with high raw anomalies in the NuPIC sense, where prediction is impossible or highly sensitive to initial conditions.
1 Like
Fergal, thanks for the interesting and informative paper. I’ve only given it a quick read, but I’ll definitely digest it further.
To summarize (for my understanding), important features of one or many systems with many dynamic and unpredictable parameters can be reconstructed through observation of one parameter over time. This reconstruction is modeled by Takens’ Theorem, which uses various delays from the time-series measured parameter to reconstruct properties of other parameters. The hypothesis is that neocortical structures like the Cortical Column use this principle to create a model of our dynamic world.
I’d also like to point to your blog writeups. They are really helpful:
http://inbits.com/2015/05/brain-universal-dynamical-systems-computer/
http://inbits.com/2015/12/new-paper-and-talk-symphonies-from-synapses/
Dave
Thanks @ddigiorg for taking a look and your kind remarks.
Yes, that is correct, but the claim is somewhat stronger than that in two ways.
First, Takens’ Theorem states that the “model” to all mathematical intents and purposes is the system from which you get the time series (formally, the manifolds are diffeomorphic), which means that anything you use the model to do (prediction, anomaly detection, detection of nonstationarity, identification of regimes) will automatically be true of the real world system.
Takens’ Theorem is a theorem (ie universally true, like Pythagoras’) for specific classes of dynamical systems, but as Sugihara et al and many others show, it still holds in large part in almost all natural settings, even where the systems and the time series do not match the criteria.
The second point is illustrated by the examples given in the two talks above. To extract the parameters of the correct embeddings, scientists need to try different settings for the dimensionalities, lags, and choices of input variables in the time series, and choose the one which gives the best prediction scores or wins on some related measure, all involving observing how nearest neighbours stay close together as you change dimensionality.
HTM and the cortex do this automatically in L4 by projecting the inputs into the very high dimensional space of the TM’s cellular SDR’s, and using its learned coincidence detection to automatically lock on to the appropriate dimensionality, lag etc. By Temporally Pooling in L2/3 over highly predicted transitions, HTM models nonstationarity in the observed system (which these guys can’t handle) and identifies the sequence of regimes the system is passing through.
As Sugihara says in the Q&A, he knows he’s figured out the parameters he needs when his model does good predictions. This is exactly how HTM and the cortex do it, but the parameters are learned and the system need not be stationary as long as its regime trajectories can be learned.
1 Like
@fergalbyrne Once again I appreciate your rich explanation about how Takens’ Theory applies to the cortex and HTM. I’ve never heard of concepts like dynamical systems, Takens’ Theorum, and diffeomorphism so this is a really unusual perspective and quite interesting!
HTM and the cortex do this automatically in L4 by projecting the inputs into the very high dimensional space of the TM’s cellular SDR’s, and using its learned coincidence detection to automatically lock on to the appropriate dimensionality, lag etc. By Temporally Pooling in L2/3 over highly predicted transitions, HTM models nonstationarity in the observed system (which these guys can’t handle) and identifies the sequence of regimes the system is passing through.
Hmm… I think I’m having trouble understanding the relation between the current HTM theory architecture with the Takens Theorem. Other than my wildly unfamiliarity with these new concepts, a portion of my confusion stems from the functionality and interaction of layers. The layered architecture you explain in your paper seems a bit different from my understanding (unless your HTM theory is a bit different than Numenta’s which might explain my confusion). However, I don’t want to further derail the thread because I haven’t had a chance to fully understand your paper. I apologies I can’t contribute anything meaningful to the discussion!
At least for me I’ve identified a glaring gap in my knowledge. I really don’t have a solid intuition about a cortical column’s architecture and it’s holding me back. I think I might do a paper or detailed diagram myself to collect my thoughts and finally start digging into some neuroscience sources. I hope you don’t mind me PMing you some questions in the coming weeks.
A side note (and attempt to contribute at least something to the conversation), I think at 3:18 and onward in this video may be indicating the L6 transformation current HTM theory. L6 transforms “what” and “where” inputs and L4 performs sensory-motor inference and prediction on that transformation. I could be wrong though, I’m not confident in my understanding just yet.
EDIT: For those who are unfamiliar with Dr. Sugihara’s work on dynamic modeling, I found an interesting article.