Deep Reinforcement Learning, HTM

Science Numenta Theory
1

Sub-sub topic: Transfer Learning or Inductive Learning
Hello,
I have been here for a bit now but I am still a newbie when it comes to learning/using HTM. I am interested in all machine learning/neuroscience/brain philosophy approaches. I have a question regarding something that I came across while reading deep reinforcement learning methods.

Context: The Deep Reinforcement Learning technique was used by Deepmind (David Silver et al) to learn various Atari video games. What I found awkward about their approach was that to learn each game, the deep reinforcement learning algorithms started from scratch. That is there was no reuse of what was previously learned.

For example, as a teenager, I played Street Fighter and then Mortal Combat. In terms of learning for me, both of those games are quite similar. Some idiosyncratic moves may not translate from one game to another but the overall logic is the same. I guess this is called transfer learning or inductive learning or recursive learning.

I think this should be a quite obvious property, if we are going to build something intelligent. That is an intelligent algorithm should be able to use what is already learned to learn something new. For example, if you learned math, you should have easier time understanding physics…

I guess my question is would HTM be natural at doing transfer learning. That is if you trained HTM to play some game and then asked it to play a very similar game, would it need to learn from scratch?

Chirag

1 Like
2

I know that @ericlaukien has experience mashing up RL techniques and HTM, so he might be interested in this topic.

(That being said, I just want to remember posters to keep this discussion about RL in the context of HTM, not a discussion about RL techniques or the RL approach in general.)

3

Thanks Matt. Also, I really like the HTM school. It’s very well done!! I
will keep the discussion in check to HTM.

Chirag

2 Likes
4

Hi Chirag,

My intuition on this is once we get hierarchy involved in the HTM algorithms, we will be able to “re-use” portions of a hierarchy as pretrained nodes that then can be used as lower levels upon which other hierarchies can be developed. But that is my guess about this. I left a similar comment in the SDR thread here.

5

Ah ok…that makes sense. That’s helpful. I read your comments in the link… I will be looking out for these feedback features. In a way this should be native to this architecture as it’s memory based. You would always have a memory of what is known as a starting point. Knowing a good starting point is a key for RL loops. In any case, I look forward to the hangout. Thanks!!

6

“Transfer learning” is a very broad phrase and I guess is equivalent to “generalisation”. To give a similarly broad answer, distributed representations should naturally support this because they will be (partially) reactivated in similar situations.

There are some interesting papers if you search for “one-shot learning”…