Sudden inability to run python files in docker containers

Engineering NuPIC
1

Running docker containers with latest numenta/nupic image and suddenly having trouble running examples I have previously run just fine. Similar error messages as follows:

/usr/local/src/nupic/examples/opf/clients/hotgym/simple# python hotgym.py
/usr/include/c++/4.8/debug/vector:346:error: attempt to subscript container
with out-of-bounds index 29, but container only holds 29 elements.

Objects involved in the operation:
sequence “this” @ 0x0x370def0 {
type = NSt7__debug6vectorIN5nupic10algorithms6Cells49InSynapseESaIS4_EEE;
}
Aborted

and

/usr/local/src/nupic/examples/network# python network_api_demo.py
spatial pooler region inputs: [‘sequenceIdIn’, ‘resetIn’, ‘topDownIn’, ‘bottomUpIn’]
spatial pooler region outputs: [‘topDownOut’, ‘spatialTopDownOut’, ‘temporalTopDownOut’, ‘bottomUpOut’, ‘anomalyScore’]
spatial pooler columns: 2048

temporal memory region inputs: [‘sequenceIdIn’, ‘resetIn’, ‘bottomUpIn’]
temporal memory region outputs: [‘lrnActiveStateT’, ‘activeCells’, ‘bottomUpOut’, ‘topDownOut’, ‘predictedActiveCells’, ‘anomalyScore’]
temporal memory columns: 2048

Writing output to network-demo-anomaly-output.csv
/usr/include/c++/4.8/debug/vector:346:error: attempt to subscript container
with out-of-bounds index 30, but container only holds 30 elements.

Objects involved in the operation:
sequence “this” @ 0x0x1dd44e0 {
type = NSt7__debug6vectorIN5nupic10algorithms6Cells49InSynapseESaIS4_EEE;
}
Aborted

Any clues? Is it possible to pull an old NuPIC image, such as 0.5.3, from which I know I can run the example python files and modify for my own experiments?

1 Like
2

I should add that on the network example the csv file IS written, but aborts ~30 lines in.

3

cc @rhyolight @subutai @mrcslws

It looks like you’re running the Debug (error-checking) build of nupic.bindings (nupic.core). What is the exact source of the container and nupic.bindings build? Please provide the URL.

You should not be hitting this error in a release build.

We’re aware of out-of-bounds index accesses at the moment that look similar to the one you have reported. See When built with -DGLIBCXX_DEBUG, c++ detects out-of-bounds access to _synapses in Segment::freeNSynapses · Issue #1014 · numenta/nupic.core-legacy · GitHub.

Would you mind reviewing When built with -DGLIBCXX_DEBUG, c++ detects out-of-bounds access to _synapses in Segment::freeNSynapses · Issue #1014 · numenta/nupic.core-legacy · GitHub and either adding your information to it or opening a new issue in nupic.core with the relevant details?

Many thanks!

Vitaly

1 Like
4

Ideally, in our CI system, we should be running tests in debug mode as well as release mode. If this bug is actually being encountered while running HotGym in debug mode, there may be lingering important bugs that are caught by debug mode but not currently addressed.

1 Like
5

Thank you for the quick response. I will continue to research and will add to the issue you linked.

Fyi…I am running the following:

OSX 10.9.5
Docker 1.11.1

I do the following:

docker pull numenta/nupic
docker run -i -t numenta/nupic /bin/bash
cd $NUPIC/examples/network
python network_api_demo.py

I did this simple operation all summer without issue. I want to understand the script well enough to import other encoders, specify different outputs, etc.

Since any container spun up is using the latest NuPIC image, this should take care of the build issues you mention, no?

6

I found the explanation for why you were able to run fine all summer, and then !! https://github.com/numenta/nupic/commit/29e1ffd5f11cfdcaa6aa9f013a2bb9e419e19489 was merged in September, and it changes Dockerfile to build Debug (error-checking) build of nupic.bindings. The Debug build compiles-in various error-checking code, including that of the C++ STL, which is what catches the nupic.core/nupic.bindings bug in this case.

I think it’s catching a real bug in nupic.core.

For now, you can suppress the reporting of this bug by changing the nupic.core CMAKE_BUILD_TYPE to Release in https://github.com/numenta/nupic/blob/master/Dockerfile.

CC @scott @rhyolight @Austin_Marshall

3 Likes
7

I added a note and a link to this topic in https://github.com/numenta/nupic.core/issues/1014.

2 Likes
8

Too late at night for me to digest, but in network_api_demo.py I changed "maxSynapsesPerSegment" (TPRegion) from 32 to 332, and it works (i.e. script does not abort/ all 2000 lines of network-demo-anomaly-output.csv are written). I am presumably still running in Debug mode.

I throw this out as a fact only–I don’t know the effect on the anomalyScore or anomalyLikelihood by increasing the maximum synapses as described above.