Reinforcment learning using NuPIC(.core)


Hi all.
I’ve been learning about HTM Theory and how to use NuPIC.core for a while. Now I want to to extend my knowledge about HTM by building a Reinforcement Learning models on NuPIC (for fun, let me know if that is impossible for now).
The first problem I met is that how can I perform reinforcement learning using NuPIC? I have read trough NuPIC.core’s source code. Yet there seems to be no way to inject any sort of signal into either a Spatial Pooler or a Temporal Pooler.
Where should I start?

Many thanks.


Hmm a tricky one, and discussed previously:

and here in a more biological detail:

There is also a recent MSc thesis which attempted to tie HTM with reinforcement learning:

I’m also interested in giving it a go, just need to find some time.



Thanks. The links gives me a a lot of infomation!
Seems like that distal connections is required to perform RL. How is that done in code? I can’t find any API that does that. (There is activateCells, but using it seems wrong…)

Many Thanks.


I agree, activate cells seems a little too core to use extensively outside the class.

If your extending the functionality of nupic/ algorithms/ temporal memory or spatial Pooler, I’d subclass from one of those. That way if you need to access the internals, you’d be keeping with the protected/public attributes.