# HTM in OpenAI Gym

**URL:** <https://discourse.numenta.org/t/htm-in-openai-gym/1007>\
**Category:** NuPIC\
**Tags:** benchmark\
**Created:** [July 15, 2016, 4:25pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007 "2016-07-15T16:25:34Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [July 15, 2016, 4:25pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/1 "2016-07-15T16:25:34Z")

</div>

Has anyone used [OpenAI Gym](https://gym.openai.com/)? I saw a [youtube comment](https://www.youtube.com/watch?v=Q1BRmv2yPh8&lc=z12vdf4bwveocria104cgrfhttqywvl4bgw) saying we should try to get an HTM agent into OpenGym, but I know nothing about it except that it uses TensorFlow or Theano as a platform.

Is anyone interested in getting an NuPIC running in OpenAI Gym?

---

<div class="post-metadata">

**Author:** ![natoromano](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@natoromano](https://discourse.numenta.org/u/natoromano)\
**Post date:** [July 15, 2016, 9:50pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/2 "2016-07-15T21:50:41Z")

</div>

Hey @rhyolight, I recently starting playing around with OpenAI Gym in the goal of applying HTM to it, but it may be a little early, and I still did not have time to push it very far. We can talk about it if you want. OpenAI Gym does not use any specific platform, and has an extremely simple API.

We can always use classic reinforcement learning algorithms, building policy or value models with HTM systems, but a more HTM-esque version of doing it would be to wait for the advances of our research on sensorimotor inference. Once we have stable code supporting this work, OpenAI Gym would be a great thing to try.

---

<div class="post-metadata">

**Author:** ![fergalbyrne](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/fergalbyrne/32/32_2.png) [@fergalbyrne](https://discourse.numenta.org/u/fergalbyrne)\
**Post date:** [July 15, 2016, 10:17pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/3 "2016-07-15T22:17:38Z")

</div>

As part of our work with OpenAI Gym, @ericlaukien has added some new tasks which you can extend via his [PGE library](https://github.com/222464/PGE). It’s a drop-in replacement which uses only Open Source components instead of OpenAI Gym’s proprietary use of 3D engines. The guys in OpenAI are currently assessing it for inclusion in future OpenAI environments. We’re looking for feedback, especially if you have examples using HTM technologies.

---

<div class="post-metadata">

**Author:** ![marion](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marion/32/491_2.png) [@marion](https://discourse.numenta.org/u/marion)\
**Post date:** [October 3, 2016, 6:41pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/4 "2016-10-03T18:41:50Z")

</div>

Reposting this [from another post](https://discourse.numenta.org/t/naive-sensorimotor-inference-engine/1460/10), in case this might be useful to others:

> @natoromano ran some experiments with HTM and OpenAI. He presented his results during an engineering meeting and his code is here: [https://github.com/natoromano/nupic.research/tree/gym/projects/gym](https://github.com/natoromano/nupic.research/tree/gym/projects/gym)

> Cheers!

> Marion

---

<div class="post-metadata">

**Author:** ![mccall.ryan](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/mccall.ryan/32/179_2.png) [@mccall.ryan](https://discourse.numenta.org/u/mccall.ryan)\
**Post date:** [January 30, 2017, 1:48am UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/5 "2017-01-30T01:48:01Z")

</div>

Hi, I was trying to get @natoromano 's openAi gym example working. I’m having trouble with a call to convertedDepolarizeCells() in extended temporal memory:

```auto
RuntimeError: CHECK FAILED: 
"std::all_of( activeCellsExternalBasal, activeCellsExternalBasal + activeCellsExternalBasalSize, [&](UInt c) { return c < numBasalInputs_; })" 
Values in activeCellsExternalBasal must be within the range [0,0).

```

this is using nupic.bindings==0.4.13. If I use the latest nupic.bindings==0.5.1, then I get a compile error in `temporal_memory_factory.py` (nupic.research) for the import:

```auto
from nupic.bindings.experimental import (
  ExtendedTemporalMemory as ExtendedTemporalMemoryCPP)`

```

so I’m a little unclear what nupic.bindings should be used to run the latest stuff in nupic.research (I’m using latest nupic of 0.5.7)?

Getting out of the weeds, the Network architecture is an observation sensor, an actionsensor, and SPRegion and an ExtendedTMRegion. The key point is that the action sensor feeds into etm “basal-ly” – so perhaps my question is could some point me to a basic example of how to run the latest nupic.research ExtendedTMRegion ?

Many thanks

Ryan

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 30, 2017, 4:18pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/6 "2017-01-30T16:18:06Z")

</div>

Perhaps @scott or @mrcslws could find time to help you out, but [remember what the research readme says](https://github.com/numenta/nupic.research#what-you-should-understand-about-this-repository). If that code has been abandoned, there may not be much interest.

---

<div class="post-metadata">

**Author:** ![Austin\_Marshall](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/austin_marshall/32/1029_2.png) [@Austin\_Marshall](https://discourse.numenta.org/u/Austin_Marshall)\
**Post date:** [January 30, 2017, 4:39pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/7 "2017-01-30T16:39:00Z")

</div>

If you’re using experimental/research code, you should be running everything in “develop” mode, rather than versioned binary releases. Getting your environment up to speed is a little trickier now, but it can be done!

I made an attempt at documenting the process for setting up your repositories to work with the newly minted nupic.research.core fork of nupic.core at [https://github.com/numenta/nupic.research/wiki/nupic.core-feature-branch-workflow](https://github.com/numenta/nupic.research/wiki/nupic.core-feature-branch-workflow). Build nupic.research.core-derived nupic.bindings from source, and install nupic from checkout.

---

<div class="post-metadata">

**Author:** ![mrcslws](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/mrcslws/32/4961_2.png) [@mrcslws](https://discourse.numenta.org/u/mrcslws)\
**Post date:** [January 30, 2017, 5:10pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/8 "2017-01-30T17:10:59Z")

</div>

The ETM is now aware of its basal/apical input dimensions, so these need to be set in the [TM params](https://github.com/natoromano/nupic.research/blob/6860d39ed39af0dd42aaf628f6659f76fc844f88/projects/gym/htm_learner.py#L58).

I see this code doesn’t set the RawSensor’s width, so I think it will use the [default](https://github.com/numenta/nupic.research/blob/86f6d2fe900c8f71becdc451156f16c4103a160c/htmresearch/regions/RawSensor.py#L39) of 2048.

So, add to the TM\_PARAMS:

```python
"basalInputWidth": 2048,

```

It would be nice if the region would figure out the width of the basal input rather than requiring you to set this. But last time I tried this I failed – the region couldn’t check the size of the input during initialization. At least partly because [it doesn’t actually receive](https://github.com/numenta/nupic.core/issues/878) the inputs and outputs during initialization.

---

<div class="post-metadata">

**Author:** ![mccall.ryan](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/mccall.ryan/32/179_2.png) [@mccall.ryan](https://discourse.numenta.org/u/mccall.ryan)\
**Post date:** [February 1, 2017, 5:40pm UTC](https://discourse.numenta.org/t/htm-in-openai-gym/1007/9 "2017-02-01T17:40:18Z")

</div>

Thanks a bunch, Marcus! That helped – I’ve got the cart-pole example running now. (using an old nupic.bindings release, I will have to update that when I figure out how to build the core)

I wonder what performance @natoromano got? I’m seeing scores in the low 30’s on average at best (number of steps the pole was kept upright).

> Number of steps at trial 57 is 16  
> Average of last 100 is 32.81

> Number of steps at trial 58 is 47  
> Average of last 100 is 33.05

FYI, I’ll be sharing the code as I go along if anyone is interested: [GitHub - ryanjmccall/htm-agents: Software agents based on HTM theory](https://github.com/ryanjmccall/htm-agents)