Hot gym prediction swarming TypeError on mac

Hi,

I have tried to get the hot gym prediction tutorial to run for quite some time.
First it was a surprise that I needed MySQL to do the swarming, now I get this output after finally installing the needed software:

This script runs a swarm on the input data (rec-center-hourly.csv) and
creates a model parameters file in the model_params directory containing
the best model found by the swarm. Dumps a bunch of crud to stdout because
that is just what swarming does at this point. You really don’t need to
pay any attention to it.

=================================================
= Swarming on rec-center-hourly data…
= Medium swarm. Sit back and relax, this could take awhile.

Generating experiment files in directory: /Users/martinlu/Downloads/nupic/nupic/examples/opf/clients/hotgym/prediction/one_gym/swarm…
Writing 312 lines…
Writing 114 lines…
done.
None
Successfully submitted new HyperSearch job, jobID=1009
Evaluated 0 models
HyperSearch finished!
Worker completion message: None

Results from all experiments:

Generating experiment files in directory: /var/folders/mn/4llfqs0j1hldlkl4m5rzyqth0000gp/T/tmp4KexIU…
Writing 312 lines…
Writing 114 lines…
done.
None
json.loads(jobInfo.results) raised an exception. Here is some info to help with debugging:
jobInfo: _jobInfoNamedTuple(jobId=1009, client=u’GRP’, clientInfo=u’’, clientKey=u’’, cmdLine=u’$HYPERSEARCH’, params=u’{“hsVersion”: “v2”, “maxModels”: null, “persistentJobGUID”: “5c73c1ca-e3b4-11e6-9608-5cf9388f078e”, “useTerminators”: false, “description”: {“inferenceArgs”: {“predictionSteps”: [1], “predictedField”: “kw_energy_consumption”}, “iterationCount”: -1, “swarmSize”: “medium”, “includedFields”: [{“fieldName”: “timestamp”, “fieldType”: “datetime”}, {“minValue”: 0.0, “fieldName”: “kw_energy_consumption”, “fieldType”: “float”, “maxValue”: 53.0}], “streamDef”: {“info”: “kw_energy_consumption”, “version”: 1, “streams”: [{“info”: “Rec Center”, “source”: “file://rec-center-hourly.csv”, “columns”: ["*"]}]}, “inferenceType”: “TemporalMultiStep”}}’, jobHash=’\ui8\xe3\xb4\x11\xe6\x83\xdd\\xf98\x8f\x07\x8e’, status=u’notStarted’, completionReason=None, completionMsg=None, workerCompletionReason=u’success’, workerCompletionMsg=None, cancel=0, startTime=None, endTime=None, results=None, engJobType=u’hypersearch’, minimumWorkers=1, maximumWorkers=4, priority=0, engAllocateNewWorkers=1, engUntendedDeadWorkers=0, numFailedWorkers=0, lastFailedWorkerErrorMsg=None, engCleaningStatus=u’notdone’, genBaseDescription=None, genPermutations=None, engLastUpdateTime=datetime.datetime(2017, 1, 26, 10, 44, 6), engCjmConnId=None, engWorkerState=None, engStatus=None, engModelMilestones=None)
jobInfo.results: None
EXCEPTION: expected string or buffer
Traceback (most recent call last):
File “swarm.py”, line 108, in
swarm(INPUT_FILE)
File “swarm.py”, line 100, in swarm
modelParams = swarmForBestModelParams(SWARM_DESCRIPTION, name)
File “swarm.py”, line 77, in swarmForBestModelParams
verbosity=1
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 271, in runWithConfig
return _runAction(runOptions)
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 212, in _runAction
returnValue = _runHyperSearch(runOptions)
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 155, in _runHyperSearch
metricsKeys=search.getDiscoveredMetricsKeys())
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 822, in generateReport
results = json.loads(jobInfo.results)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”, line 338, in loads
return _default_decoder.decode(s)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

Here are the steps to replicate, just paste line by line:

sudo pip install virtualenv
virtualenv nupic
cd nupic
./bin/pip install --upgrade pip
./bin/pip install nupic
git clone https://github.com/numenta/nupic.git
cd nupic/examples/opf/clients/hotgym/prediction/one_gym/
curl https://download.docker.com/mac/stable/Docker.dmg -o docker.dmg && open docker.dmg
docker run --name nupic-mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -p 3306:3306 -d mysql:5.6
…/…/…/…/…/…/…/bin/python …/…/…/…/…/swarm/test_db.py
…/…/…/…/…/…/…/bin/python swarm.py

I also tried to run the simple script:

…/…/…/…/…/…/bin/python hotgym.py

but that gave the error:

ERR: Unknown parameter ‘boostStrength’ for region ‘SP’ of type ‘py.SPRegion’

As you can see there are no modifications to any of the code.
Any ideas on how do I get the tutorial to run?

1 Like
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

That error seems to indicate that the the JSON swarm description file you are using is not valid JSON. Can you run in through a JSON validator and check?

Yes I have and it validates perfectly.
It is the swarm definition directly from the repository:

Try this before we move ahead? There may be a mismatch between your installed nupic version and the code you’re running from the git checkout.

Sorry, still no luck.
I did this:

git fetch origin

Where origin points to the remote on github.
and then

git checkout tags/0.5.7

and finally ran

…/…/…/…/…/…/…/bin/python swarm.py

which resulted in the same error

This script runs a swarm on the input data (rec-center-hourly.csv) and
creates a model parameters file in the model_params directory containing
the best model found by the swarm. Dumps a bunch of crud to stdout because
that is just what swarming does at this point. You really don’t need to
pay any attention to it.

=================================================
= Swarming on rec-center-hourly data…
= Medium swarm. Sit back and relax, this could take awhile.

Generating experiment files in directory: /Users/martinlu/Downloads/nupic/nupic/examples/opf/clients/hotgym/prediction/one_gym/swarm…
Writing 312 lines…
Writing 114 lines…
done.
None
Successfully submitted new HyperSearch job, jobID=1001
Evaluated 0 models
HyperSearch finished!
Worker completion message: None

Results from all experiments:

Generating experiment files in directory: /var/folders/mn/4llfqs0j1hldlkl4m5rzyqth0000gp/T/tmprficR7…
Writing 312 lines…
Writing 114 lines…
done.
None
json.loads(jobInfo.results) raised an exception. Here is some info to help with debugging:
jobInfo: _jobInfoNamedTuple(jobId=1001, client=u’GRP’, clientInfo=u’’, clientKey=u’’, cmdLine=u’$HYPERSEARCH’, params=u’{“hsVersion”: “v2”, “maxModels”: null, “persistentJobGUID”: “e4af43c0-e3e6-11e6-8add-5cf9388f078e”, “useTerminators”: false, “description”: {“inferenceArgs”: {“predictionSteps”: [1], “predictedField”: “kw_energy_consumption”}, “iterationCount”: -1, “swarmSize”: “medium”, “includedFields”: [{“fieldName”: “timestamp”, “fieldType”: “datetime”}, {“minValue”: 0.0, “fieldName”: “kw_energy_consumption”, “fieldType”: “float”, “maxValue”: 53.0}], “streamDef”: {“info”: “kw_energy_consumption”, “version”: 1, “streams”: [{“info”: “Rec Center”, “source”: “file://rec-center-hourly.csv”, “columns”: ["*"]}]}, “inferenceType”: “TemporalMultiStep”}}’, jobHash=’\xe4\xb0\xe2\xf3\xe3\xe6\x11\xe6\xa8\xb5\\xf98\x8f\x07\x8e’, status=u’notStarted’, completionReason=None, completionMsg=None, workerCompletionReason=u’success’, workerCompletionMsg=None, cancel=0, startTime=None, endTime=None, results=None, engJobType=u’hypersearch’, minimumWorkers=1, maximumWorkers=4, priority=0, engAllocateNewWorkers=1, engUntendedDeadWorkers=0, numFailedWorkers=0, lastFailedWorkerErrorMsg=None, engCleaningStatus=u’notdone’, genBaseDescription=None, genPermutations=None, engLastUpdateTime=datetime.datetime(2017, 1, 26, 16, 45, 50), engCjmConnId=None, engWorkerState=None, engStatus=None, engModelMilestones=None)
jobInfo.results: None
EXCEPTION: expected string or buffer
Traceback (most recent call last):
File “swarm.py”, line 109, in
swarm(INPUT_FILE)
File “swarm.py”, line 101, in swarm
modelParams = swarmForBestModelParams(SWARM_DESCRIPTION, name)
File “swarm.py”, line 78, in swarmForBestModelParams
verbosity=0
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 271, in runWithConfig
return _runAction(runOptions)
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 212, in _runAction
returnValue = _runHyperSearch(runOptions)
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 155, in _runHyperSearch
metricsKeys=search.getDiscoveredMetricsKeys())
File “/Users/martinlu/Downloads/nupic/lib/python2.7/site-packages/nupic/swarming/permutations_runner.py”, line 822, in generateReport
results = json.loads(jobInfo.results)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”, line 338, in loads
return _default_decoder.decode(s)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

Based on these lines it looks like no results are found:

engCjmConnId=None, engWorkerState=None, engStatus=None, engModelMilestones=None)
jobInfo.results: None
EXCEPTION: expected string or buffer

Can you verify what version of NuPIC you have installed by running this crazy long command:

python -c 'import pkg_resources;n=pkg_resources.get_distribution("nupic");b=pkg_resources.get_distribution("nupic.bindings");print n.project_name, n.version;print b.project_name, b.version'

It should print something like this:

nupic 0.5.7
nupic.bindings 0.4.13

That crazy long command produced:

nupic 0.5.7
nupic.bindings 0.4.13

I don’t get it. This error is something we usually see when there is an newer version of code running against an older version of NuPIC installed on your system:

ERR: Unknown parameter 'boostStrength' for region 'SP' of type 'py.SPRegion'

This was an API change in December. By all accounts, it still looks like your local checkout is newer than your nupic installation. I can’t see any other reason this would happen.

I now remade the procedure listed in my first post with the only exception of only checking out the 0.5.7 tag from github, released on 28 Nov 2016.
The binary install is from https://pypi.python.org/pypi/nupic/0.5.7 released date 2016-11-28.
Now the

hotgym.py

in

nupic/examples/opf/clients/hotgym/simple

produces

INFO:main:After 100 records, 1-step altMAPE=23.183145
INFO:main:After 200 records, 1-step altMAPE=21.548877
INFO:main:After 300 records, 1-step altMAPE=21.227594
INFO:main:After 400 records, 1-step altMAPE=20.686270
INFO:main:After 500 records, 1-step altMAPE=20.417234
INFO:main:After 600 records, 1-step altMAPE=20.852339
INFO:main:After 700 records, 1-step altMAPE=20.907660
INFO:main:After 800 records, 1-step altMAPE=21.137106
INFO:main:After 900 records, 1-step altMAPE=20.875763
INFO:main:After 1000 records, 1-step altMAPE=20.789707

But the swarming do still not work.
By running

swarm.py

in

nupic/examples/opf/clients/hotgym/prediction/one_gym

Received the same error when swarming.

What ideas do you have for moving forward with swarming?

I am diving into the code to try to figure out why object (jobInfo.results) can be None

This is a red herring, it just means that the HTM run failed:

https://github.com/numenta/nupic/issues/1815

It usually means that the NuPIC run was misconfigured or that the incoming data was not the right type. Try running the simple hotgym example as described here and see if that returns an error.

1 Like

Hi Matt,

Yes, running the simple hotgym example works just great.
Then I get the out put:

INFO:main:After 100 records, 1-step altMAPE=23.183145
INFO:main:After 200 records, 1-step altMAPE=21.548877
INFO:main:After 300 records, 1-step altMAPE=21.227594
INFO:main:After 400 records, 1-step altMAPE=20.686270
INFO:main:After 500 records, 1-step altMAPE=20.417234
INFO:main:After 600 records, 1-step altMAPE=20.852339
INFO:main:After 700 records, 1-step altMAPE=20.907660
INFO:main:After 800 records, 1-step altMAPE=21.137106
INFO:main:After 900 records, 1-step altMAPE=20.875763
INFO:main:After 1000 records, 1-step altMAPE=20.789707

Sounds like I need to compile NuPIC from source in order to get Swarming to work at all?
Or is there a swarm tutorial version that works with the binary install?

That verifies that NuPIC is running, so it seems to have been installed properly. The next thing to check would be that the incoming lines of data to the swarm match the data fields described in the swarm description file. Can you copy and paste the swarm description JSON into gist? Also a sample of your your data file would help a lot (headers and a few rows of data).

This can happen sometimes if somewhere in your data file you have an empty field, where an int was expected.

Great,

I am running the hot_gym tutorial exactly as available in your git, no changes at all.

I want to get the default setup running first, before starting dissecting it to do my own stuff.

That is strange… I just installed nupic via pip install nupic==0.5.7 in a fresh virtualenv and ran examples/opf/clients/hotgym/prediction/one_gym/swarm.py without a problem.

Are you running the script like this?

python swarm.py

Or using your default system python like this?

./swarm.py

Hmm, strange indeed.

I run exactly these commands:

sudo pip install virtualenv
virtualenv nupic
cd nupic
./bin/pip install --upgrade pip
./bin/pip install nupic
git clone —branch 0.5.7 https://github.com/numenta/nupic.git
cd nupic/examples/opf/clients/hotgym/prediction/one_gym/
curl https://download.docker.com/mac/stable/Docker.dmg -o docker.dmg && open docker.dmg
docker run --name nupic-mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -p 3306:3306 -d mysql:5.6
…/…/…/…/…/…/…/bin/python …/…/…/…/…/swarm/test_db.py
…/…/…/…/…/…/…/bin/python swarm.py

@Austin_Marshall @vkruglikov Hey do you guys know any reason the docker invocation mentioned by @martin above would be causing this problem? It’s the only thing I can think of.

@rhyolight @Austin_Marshall @vkruglikov, just wanted to check in if you had any ideas or news on why the swarming fails?
Could it be the Docker invocation?

Martin, can you try installing outside of Docker? Like in a virtualenv?

Everything except MySQL is installed in a virtualenv outside of docker.
Might it not be something with the tutorial code?