Why is HTM Ignored by Google DeepMind?



This is the essence of all online learning algorithms, DL and HTM included. The training set is the continuously arriving input stream. When the statistics of the input stream change, the model adjusts to fit the new patterns. DQN is one example of learning online*, and if the task domain changes (let’s say you change how rewards are delivered) then the model will adapt to that. In fact, this is the concept behind a technique in RL called “reward shaping”, where you give the network an easier task to solve at first in order to learn useful preliminary behaviors, and then progressively increase the difficulty of the tasks in order to coax it into learning more complex skills.

So HTM has no monopoly on adapting to a changing world. It does it in a much more biologically appealing way, certainly, and as a result it has the potential to explain the brain, which most deep networks do not. But the capability to adapt is not beyond mainstream machine learning. As I mentioned above however, HTM has a silver bullet that should in principle help it adapt more quickly to a changing world, which is the sparsity of its learning updates. So each HTM synaptic update is less likely to interfere with the previous things you’ve learned than would a dense update in a standard deep network, and you can be more aggressive with your updates as a result, therefore learning faster.

I really do view the benefits of sparsity as the sole performance advantage of HTM. But that advantage is beyond massive, so it is certainly worth exploiting.

(*DQN is a bit of a special case because of experience replay: when the task changes the replay buffer will get stale, full of data about the world as it used to be, rather than it is now. A better example would be A3C, the new-ish hotness in reinforcement learning, which does on-policy updates using the present experience instead of a replay buffer, in a much more “online” way.)


Really enjoying this discussion, and very useful to me as a relative newcomer to HTM. Thank you!

I just wanted to pick up on something that @Charles_Rosenbauer mentioned in an earlier post:

Does anyone have any thoughts about this, or how we might approach a solution?

… that is, in addition to this post by @Charles_Rosenbauer which deals with some of the biological aspects.


This is one area I have been exploring recently, and there are others on the forum who are actively working on it as well. To me this is the obvious next function to understand after sensory-motor integration (or rather part of it IMO), so I wouldn’t be surprised if Numenta begins tackling it themselves in the not too distant future. You can follow my project on this thread. Lately I’ve I got a little side-tracked writing my own implementation of semantic folding to try and understand that concept better, but I’ll be getting back to RL soon.


“A significant component of the DQN training algorithm is a mechanism called experience replay [5]. Transitions experienced from interacting with the environment are stored in the experience replay memory. These transitions are then uniformly sampled from to train on in an offline manner. From a theoretical standpoint this breaks the strong temporal correlations that would affect learning online.”-torch Dueling Deep Q-Networks

That does not appear to be online.

“In both the brain, and in deep networks, learning is not immediate. HTM abstracts the forming and strengthening of synapses into a single step process, but it’s actually a multiple-step chain of biochemical interactions that requires many repetitions of the pattern you want to learn. So each time a neuron sees a pattern, its connections are only updated by a small amount, and many repetitions are required to fully solidify the pattern detector. Deep networks are trained in an analogous way, where each presentation of a pattern only updates the synapses a small amount.”

Within seconds brain tissue experiences structural changes that can affect performance. Even without a hyppocampus short term memory works, and learning can occur, it is just that it is quickly forgotten and not permanent. The issue with the need for the hippocampus probably has to do with metaplasticity rules in the brain. If the higher areas have neurons active over longer periods of time as compared to areas lower in the hierarchy, they must have different requirements to make changes to permanence. Metaplasticity can supposedly solve catastrophic forgetting and drastically improve the memory capacity of a neural system.

As regards online learning, the brain can learn while being active in the environment and quickly even within seconds change and adapt to novel information, without being taken offline, that is it can experience changes even drastic ones without interrupting waking activity. True some cases it takes time to improve performance, but in the simpler examples even drastic performance is possible in seconds.

“There are 30 million seconds in a year, and if we assume we process input at 10Hz or more, that’s hundreds of millions of training examples per year, and humans don’t become useful until after more than a decade of training.”

Those are not millions of unique labelled data, a baby may spends most hours asleep, the few hours awake it may spend lots of time looking at one or two toys perhaps even a blank wall. The number of unique voices and sentences can be quite limited. Yet in a few years it will eclipse most anything, and some can even do advanced mathematics and multiple languages.


As far as I know, @Paul_Lamb and me are working on it for quite some time among the HTM forum. You can just click on his name and check out the discussions he is mainly involved in :slight_smile: Other than that there are the works of Otahal [1] and Gomez [2] which involve coupling HTM with reinforcement learning. So we are hopefully getting there :slight_smile:

I will present my MS thesis in 10 days named “Hierarchical Temporal Memory Based Autonomous Agent For Partially Observable Video Game Environments”. A real-time and online HTM architecture combined with TD(Lambda) and directed by the research on computational models of basal ganglia. It can solve simple navigation tasks in a 3D video game environment with some actual results via its visual sensor. I will for sure share it here when it is presented.

[1] https://dspace.cvut.cz/bitstream/handle/10467/21143/F3-DP-2014-Otahal-Marek-prace.pdf
[2] http://studentnet.cs.manchester.ac.uk/resources/library/3rd-year-projects/2016/antonio.sanchezgomez.pdf


I can’t wait to see it!


It works, right? I mean many people above have emphasized on more practical reasons but even if you neglect the pragmatics, HTM is capable of results even at this stage. When compared to cutting edge machine learning techniques, HTM is supposed to perform worse at a single problem but also supposed to perform with considerably great accuracy over multiple types of streaming data, inference types and problems without any parameter tweaks to its overall model, which I believe ML cannot do; HTM can, even at this point. Again, not hating on ML, even though I’d like to, because it gets marketed as AI. I felt as though you are not being objective and fair about this.
I also feel as though some noteworthy properties of HTM are singularly added to ML algorithms and they do work there. Not saying that those ideas are taken from HTM but that those principles are already working in HTM.
Apologies if it seems rude. I think it deserves clarification.


I see a lot of people saying that HTM doesn’t work, would anyone care to explain why DARPA built their own version in 2015 budget including plans in 2016 for dedicated hardware on those neural chips? Maybe we are getting the civilian version :sweat_smile:

“Develop a hierarchical temporal memory (HTM) algorithm including new data representations, low precision and ability to adapt and scale.”


I think the confusion is with the use of the term “works”.

A metaphor would be like if your goal was the commercial transport of > 50 passengers by airliner. Classical ML Techniques would be like propeller airplanes, and HTM would be a Jet engine in development stages (not yet shipped with commercial airliners). Everybody “knows” the potential of Jets however more research and development are needed to make it viable for everyday application.

In HTM’s case, a lot more development is needed, but its development is very accelerated; potential is long-range; and expected to deliver on its promise.

The term “works” is not a valid one. HTM “works” but not to the level of its potential just yet. But it’s that GREAT potential that thrusts it into comparison with techniques that are more applicable at the time.

It’s like someone has the idea of a “wheel” and immediately you can see the (conceptual) improvement over “skids”, but then someone asks immediately if it works! Well you first have to make the thing before you can judge that! :slight_smile: HTM is more “developed” than that, but the same mechanism applies.