I’ve been trying to rationalize using HTM as a world model for an AI for the RTS game Starcraft: Broodwar. If you’re not familiar, Broodwar has a lively AI/Bot scene (which you can learn about here: https://www.starcraftai.com/wiki/Main_Page) that uses innovative tech like neural networks in some bots for build orders and high level reactions.
I was thinking that a HTM could be interesting to use in conjunction with other general NN structure and AI features to have as a world model for things like battle tactics against the enemy player and build order strategies, but I would really appreciate your guys’ insight about how this system can be used in the best way possible, because I do have some issues with just jumping straight into using HTMs for this.
I would want the AI to learn in an unsupervised way for the first parts of training. Instead of traditionally just trying to win It would explore it’s environment and internalize how the world works: “I can build drones from larvae, If you attack a unit it will die once it loses all it’s health. You need a lair to build a spire.” This could be input for a traditional reinforcement AI or something of the like where they can use the prediction aspect of the HTM as a world model that can be used for prediction of the world and a deep understanding of what It can/can’t do and what it’s environment reacts to.
I was also wondering if there was a structure to make an AI that can integrate the unsupervised learning and some sort of action of it’s own so it can use action to explore it’s environment instead of just predicting from input.
But yeah, I would really appreciate your guys’ input! Feel free to ask more from me too if I didn’t explain some things enough. Ciao.
I can’t help you actually make use of HTM & neuroscience, but I definitely agree with your sentiment. Starcraft is an interesting playground for studying AI. It has a lot of complexity in the tactics and strategies but is relatively simple in terms of sensory processing.
One of the open problems with the HTM theory is that there isn’t a theory for how motor outputs are generated, which makes it had to actually control a robot using it.
I thought that the final rounds of this years SSCAIT tournament really demonstrated the importance of learning on-line, when monster played the same cheesy trick on stardust 7 times in a row. Online learning is something that basically all neurons are capable of doing.
Yeah I totally agree. To your last point I was reminded about Alphastar in one of it’s first publicly available matches that were broadcast against a good player, I couldn’t remember exactly but they were playing Protoss against Alphastar and he kept continuously immortal dropping Alphastar then leaving, which Alphastar had no memory or new strategy to counter and the human won against the AI very easily by exploiting it. Which I don’t think online learning would struggle from at all. It seems like a very important aspect to strategical thinking.
I don’t exactly know what the most modern HTM theory actually is or Thousand Brains so forgive me for that. But I would probably like a system the most where it can be completely run in parallel with it’s training, so there would be no training steps but it’s constant exposure to the medium it’s playing in and it learning it like that.
The motor system would be of my main interest though, how does the AI used the HTM part in conjunction with the other parts of the brain. I was thinking a sort of system like this:
(World Input) → (Encoding of some sort, Maybe a CONVnet of some sort that is trained on Starcraft) → (HTM) → (Some sort of “Agent” net that takes the predictions of the HTM and uses it to make actions in the world)
This would be a constant feedback loop where the bot would be constantly learning the world and also making actions in it. I don’t know what would be the best system though.