Could Unity (the game engine) use HTM

I am currently using Unity to learn ML. I am using their “ML_Agents” system to learn to play my Unity game.

Unity currently uses this Reinforcement system

It would be great to make HTM an alternative.

How could we go about getting Unity to adopt HTM?


1 Like

HTM is not like a reinforcement system, so it would not be a replacement for that ml-agents library, but yes you can use it in Unity. We run sensorimotor experiments and demos in unity.

Do you have a link to your sensorimotor experiments and demos in unity.

Just to clarify, I never meant “an alternative reinforcement system”, merely an additional ML system. Sorry about that.

Hi, Matt!

Please, could you tell more about integration of Nupic/Python into Unity/C#?
I found this old related video: .
Did something change since then?

And, can you build an app, that use HTM, in Unity ? Or for now it is only possible to run experiments in Editor Mode?

1 Like

I only looked at Unity with its AI APIs briefly but it looks like I could run an external server that is python with TensorFlow-- I think. But since I work with Digital Humans, I would have real-time augmented avatar and I don’t think TensorFlow will do it. So am looking at HTM or some variation of that.

It can be done in Second Life and people has used it for machinima but as real avatars. Some research have done Bots where they stand around and bump into walls.

Now UE4 (Unreal) has been active in real-time high resolution character rendering for a long time now but it use real actors with its motion-control tracking hardware. I don’t want to use motion tracking.

Right now I am using models from DAZ3D; its nice because I have everything I need and I don’t have to do scanning. But the problem is AI. The models don’t do motion unless I build the motions. What I would like to do is have a video of one or two persons doing something, then the AI figures what are doing and create an animation file. Then I could go to Unity or UE4 or anything that is multiuser server and recreate the animations. I could do it in Second Life because the animation can last for 30 seconds and you can run through the animations a lot-- over and over and it easy for me to chain the animations – sort of.

Could I use HTM as a starting place?