Reward hacking in simple HTM agents (using OpenAI Gym)

[marty1885]

@MaxLee

They would learn trends and general directions

Asking a SP to learn a trend is kinda out of scope, I guess?

Also, if the output of the entire primed is just the input (of the next time step). Wouldn’t it be the same to just pass the input to the motors?

@MaxLee

it manages to learn how to modulate/coordinate these different signals depending on different needs

This reminded me of a glaring problem of HTM that have bugged me for a while. How can signal be combined in a meaningful way? Simply concatenating SDRs together then send the result into a SP won’t cut because SP topology stops signals from being merged and learned efficiently. Stacking multiples SPs also doesn’t look like a viable solution due to run-away hebian learning. And there seems no other ways to integrate signals from multiple sources. Deep Learning uses gradients to guide the integration, but HTM’s local learning rules prevents it.

@MaxLee

Yes, in the both the initial input pool, as well as by the state-change pools above it. Whether or not those predictions are used to produce output would be decided by the modulator (our thalamus) in this architecture. These, in conjunction with our thalamus, learn to guide the whole system as to which output(s) should be activated to correct a given state in our system (hunger, pain, fatigue, etc.)

Hmm may you explain how this might work? I don’t see a path to actually implement this (given the issue described above). I think I’m thinking too computer science direction. May you inspire?