Reinforcment learning using NuPIC(.core)

[marty1885]

Hi all.
I’ve been learning about HTM Theory and how to use NuPIC.core for a while. Now I want to to extend my knowledge about HTM by building a Reinforcement Learning models on NuPIC (for fun, let me know if that is impossible for now).
The first problem I met is that how can I perform reinforcement learning using NuPIC? I have read trough NuPIC.core’s source code. Yet there seems to be no way to inject any sort of signal into either a Spatial Pooler or a Temporal Pooler.
Where should I start?

Many thanks.

[momiji]

Hmm a tricky one, and discussed previously:

A testcase of HTM with reinforcement learning

The problem that you’re describing is a sensorimotor and goal-oriented behavior problem. Sensorimotor means that the snake must model changes that it observes that are caused by its own actions, as opposed to external changes that it passively observes. Goal-oriented behavior means the snake must additionally drive motor commands in pursuit of some goals. In your case, there are positive and negative signals and the snake must learn to drive sequences of motor commands to get to the positive r…

and here in a more biological detail:

Exploring Reinforcement Learning in HTM

I have been working on a test to implement reinforcement learning using HTM concepts. I am still debugging some problems (with the implementation, not necessarily with the theory itself) so I am not ready to release the code just yet. But I thought I would discuss the approach I am using, and maybe get some feedback and ideas. There are a few aspects which are not biologically feasible, but I am hoping some useful information might be distilled from this exercise. When I refer to Layer X “pro…

There is also a recent MSc thesis which attempted to tie HTM with reinforcement learning:

http://studentnet.cs.manchester.ac.uk/resources/library/3rd-year-projects/2016/antonio.sanchezgomez.pdf

I’m also interested in giving it a go, just need to find some time.

M

[marty1885]

Thanks. The links gives me a a lot of infomation!
Seems like that distal connections is required to perform RL. How is that done in code? I can’t find any API that does that. (There is activateCells, but using it seems wrong…)

Many Thanks.

[SimLeek]

I agree, activate cells seems a little too core to use extensively outside the class.

If your extending the functionality of nupic/ algorithms/ temporal memory or spatial Pooler, I’d subclass from one of those. That way if you need to access the internals, you’d be keeping with the protected/public attributes.