Problem with running the swarm in the HotGym example on windows


#1

After installing Nupic on my Windows OS i tried to follow the HotGym example but i am running into problems when running swarm.py

this is the output i get when i run swarm.py:

D:\a\nupic hot gym>python swarm.py
This script runs a swarm on the input data (rec-center-hourly.csv) and
creates a model parameters file in the `model_params` directory containing
the best model found by the swarm. Dumps a bunch of crud to stdout because
that is just what swarming does at this point. You really don't need to
pay any attention to it.

=================================================
= Swarming on rec-center-every-15m-large data...
= Medium swarm. Sit back and relax, this could take awhile.
=================================================
Generating experiment files in directory: D:\a\nupic hot gym\swarm...
Writing  314 lines...
Writing  114 lines...
done.
None
Successfully submitted new HyperSearch job, jobID=1011
<jobID: 1011> 5  models finished [success: 0; eof: 0; stopped: 0; killed: 0; ERROR: 5; ORPHANED: 0; unknown: 0]
ERROR MESSAGE: Exception occurred while running model 2536: IOError(2, 'No such file or directory') (<type 'exceptions.IOError'>)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nupic\swarming\utils.py", line 435, in runModelGivenBaseAndParams
    (completionReason, completionMsg) = runner.run()
  File "C:\Python27\lib\site-packages\nupic\swarming\ModelRunner.py", line 237, in run
    maxTimeout=readTimeout)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 202, in __init__
    bookmark, firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 298, in _openStream
    firstRecord=firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\file_record_stream.py", line 175, in __init__
    self._file = open(self._filename, self._mode)
IOError: [Errno 2] No such file or directory: u'D:\\a\\nupic hot gym\\ter-every-15m-large.csv'

##>> UPDATED WORKER STATE:
{   u'activeSwarms': [   u'modelParams|sensorParams|encoders|kw_energy_consumption',
                         u'modelParams|sensorParams|encoders|timestamp_dayOfWeek',
                         u'modelParams|sensorParams|encoders|timestamp_timeOfDay',
                         u'modelParams|sensorParams|encoders|timestamp_weekend'],
    u'blackListedEncoders': [],
    u'lastGoodSprint': None,
    u'lastUpdateTime': 1511286337.33,
    u'searchOver': False,
    u'sprints': [   {   u'bestErrScore': None,
                        u'bestModelId': None,
                        u'status': u'active'}],
    u'swarms': {   u'modelParams|sensorParams|encoders|kw_energy_consumption': {   u'bestErrScore': None,
                                                                                   u'bestModelId': None,
                                                                                   u'sprintIdx': 0,
                                                                                   u'status': u'active'},
                   u'modelParams|sensorParams|encoders|timestamp_dayOfWeek': {   u'bestErrScore': None,
                                                                                 u'bestModelId': None,
                                                                                 u'sprintIdx': 0,
                                                                                 u'status': u'active'},
                   u'modelParams|sensorParams|encoders|timestamp_timeOfDay': {   u'bestErrScore': None,
                                                                                 u'bestModelId': None,
                                                                                 u'sprintIdx': 0,
                                                                                 u'status': u'active'},
                   u'modelParams|sensorParams|encoders|timestamp_weekend': {   u'bestErrScore': None,
                                                                               u'bestModelId': None,
                                                                               u'sprintIdx': 0,
                                                                               u'status': u'active'}}}
<jobID: 1011> 7  models finished [success: 0; eof: 0; stopped: 0; killed: 0; ERROR: 7; ORPHANED: 0; unknown: 0]
ERROR MESSAGE: Exception occurred while running model 2661: IOError(2, 'No such file or directory') (<type 'exceptions.IOError'>)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nupic\swarming\utils.py", line 435, in runModelGivenBaseAndParams
    (completionReason, completionMsg) = runner.run()
  File "C:\Python27\lib\site-packages\nupic\swarming\ModelRunner.py", line 237, in run
    maxTimeout=readTimeout)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 202, in __init__
    bookmark, firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 298, in _openStream
    firstRecord=firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\file_record_stream.py", line 175, in __init__
    self._file = open(self._filename, self._mode)
IOError: [Errno 2] No such file or directory: u'D:\\a\\nupic hot gym\\ter-every-15m-large.csv'

####>> UPDATED JOB RESULTS:
{   u'absoluteFieldContributions': {   u'kw_energy_consumption': nan,
                                       u'timestamp_dayOfWeek': nan,
                                       u'timestamp_timeOfDay': nan,
                                       u'timestamp_weekend': nan},
    u'fieldContributions': {   u'kw_energy_consumption': nan,
                               u'timestamp_dayOfWeek': nan,
                               u'timestamp_timeOfDay': nan,
                               u'timestamp_weekend': nan}} (elapsed time: 3.016 secs)
Evaluated 7 models
HyperSearch finished!
Worker completion message: E10002: Exiting due to receiving too many models failing from exceptions (6 out of 6).
Model Exception: Exception occurred while running model 2661: IOError(2, 'No such file or directory') (<type 'exceptions.IOError'>)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nupic\swarming\utils.py", line 435, in runModelGivenBaseAndParams
    (completionReason, completionMsg) = runner.run()
  File "C:\Python27\lib\site-packages\nupic\swarming\ModelRunner.py", line 237, in run
    maxTimeout=readTimeout)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 202, in __init__
    bookmark, firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 298, in _openStream
    firstRecord=firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\file_record_stream.py", line 175, in __init__
    self._file = open(self._filename, self._mode)
IOError: [Errno 2] No such file or directory: u'D:\\a\\nupic hot gym\\ter-every-15m-large.csv'


Results from all experiments:
----------------------------------------------------------------
Generating experiment files in directory: c:\users\samu\appdata\local\temp\tmp5najcg...
Writing  314 lines...
Writing  114 lines...
done.
None
Traceback (most recent call last):
  File "swarm.py", line 113, in <module>
    swarm(INPUT_FILE)
  File "swarm.py", line 105, in swarm
    modelParams = swarmForBestModelParams(SWARM_DESCRIPTION, name)
  File "swarm.py", line 82, in swarmForBestModelParams
    verbosity=0
  File "C:\Python27\lib\site-packages\nupic\swarming\permutations_runner.py", line 271, in runWithConfig
    return _runAction(runOptions)
  File "C:\Python27\lib\site-packages\nupic\swarming\permutations_runner.py", line 212, in _runAction
    returnValue = _runHyperSearch(runOptions)
  File "C:\Python27\lib\site-packages\nupic\swarming\permutations_runner.py", line 155, in _runHyperSearch
    metricsKeys=search.getDiscoveredMetricsKeys())
  File "C:\Python27\lib\site-packages\nupic\swarming\permutations_runner.py", line 819, in generateReport
    raise Exception(jobInfo.workerCompletionMsg)
Exception: E10002: Exiting due to receiving too many models failing from exceptions (6 out of 6).
Model Exception: Exception occurred while running model 2661: IOError(2, 'No such file or directory') (<type 'exceptions.IOError'>)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\nupic\swarming\utils.py", line 435, in runModelGivenBaseAndParams
    (completionReason, completionMsg) = runner.run()
  File "C:\Python27\lib\site-packages\nupic\swarming\ModelRunner.py", line 237, in run
    maxTimeout=readTimeout)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 202, in __init__
    bookmark, firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\stream_reader.py", line 298, in _openStream
    firstRecord=firstRecordIdx)
  File "C:\Python27\lib\site-packages\nupic\data\file_record_stream.py", line 175, in __init__
    self._file = open(self._filename, self._mode)
IOError: [Errno 2] No such file or directory: u'D:\\a\\nupic hot gym\\ter-every-15m-large.csv'

and this is the swarm file i am using (straight from the hotgym github)


import os
import pprint

import logging
logging.basicConfig()

from nupic.swarming import permutations_runner
from swarm_description import SWARM_DESCRIPTION

INPUT_FILE = "rec-center-every-15m-large.csv"
DESCRIPTION = (
  "This script runs a swarm on the input data (rec-center-hourly.csv) and\n"
  "creates a model parameters file in the `model_params` directory containing\n"
  "the best model found by the swarm. Dumps a bunch of crud to stdout because\n"
  "that is just what swarming does at this point. You really don't need to\n"
  "pay any attention to it.\n"
  )



def modelParamsToString(modelParams):
  pp = pprint.PrettyPrinter(indent=2)
  return pp.pformat(modelParams)



def writeModelParamsToFile(modelParams, name):
  cleanName = name.replace(" ", "_").replace("-", "_")
  paramsName = "%s_model_params.py" % cleanName
  outDir = os.path.join(os.getcwd(), 'model_params')
  if not os.path.isdir(outDir):
    os.mkdir(outDir)
  # Create an __init__.py so the params are recognized.
  initPath = os.path.join(outDir, '__init__.py')
  open(initPath, 'a').close()
  outPath = os.path.join(os.getcwd(), 'model_params', paramsName)
  with open(outPath, "wb") as outFile:
    modelParamsString = modelParamsToString(modelParams)
    outFile.write("MODEL_PARAMS = \\\n%s" % modelParamsString)
  return outPath



def swarmForBestModelParams(swarmConfig, name, maxWorkers=4):
  outputLabel = name
  permWorkDir = os.path.abspath('swarm')
  if not os.path.exists(permWorkDir):
    os.mkdir(permWorkDir)
  modelParams = permutations_runner.runWithConfig(
    swarmConfig,
    {"maxWorkers": maxWorkers, "overwrite": True},
    outputLabel=outputLabel,
    outDir=permWorkDir,
    permWorkDir=permWorkDir,
    verbosity=0
  )
  modelParamsFile = writeModelParamsToFile(modelParams, name)
  return modelParamsFile



def printSwarmSizeWarning(size):
  if size is "small":
    print "= THIS IS A DEBUG SWARM. DON'T EXPECT YOUR MODEL RESULTS TO BE GOOD."
  elif size is "medium":
    print "= Medium swarm. Sit back and relax, this could take awhile."
  else:
    print "= LARGE SWARM! Might as well load up the Star Wars Trilogy."



def swarm(filePath):
  name = os.path.splitext(os.path.basename(filePath))[0]
  print "================================================="
  print "= Swarming on %s data..." % name
  printSwarmSizeWarning(SWARM_DESCRIPTION["swarmSize"])
  print "================================================="
  modelParams = swarmForBestModelParams(SWARM_DESCRIPTION, name)
  print "\nWrote the following model param files:"
  print "\t%s" % modelParams



if __name__ == "__main__":
  print DESCRIPTION
  swarm(INPUT_FILE)

one thing i noticed is that in the error message where it tells me the path of the file its trying to read the filename has // instead of the regular /

has somebody here run into similar problems before and could help me figure it out?

*edit: removed &#65279 i added manually because of formatting issues


#2

This file path looks wrong.


#3

okay in the swarm.py file i speficied the path like such: INPUT_FILE = “D:\a\nupic hot gym\rec-center-every-15m-large.csv”

now the error i get looks like this: IOError: [Errno 2] No such file or directory: u’D:\ \a\ \nupic hot gym\ \ter-every-15m-large.csv’

i had to add spaces between the \ or the css formatting of this reply form makes them dissapear

i have no idea where the double \ and the u’ are coming from


#4

It seems you new to python, right? I suggest you look up how to read and write files in python and practice with some local files to make sure you can do it. I think is still just a simple file path error.


#5

I forgot to specify the file path in swarm_description.py, it is working now.
My bad sorry for wasting your time on such a simple error.
Yes i am new to python i am learning it because i want to play around with nupic.


#6

The same problem is happening on my MAC system as I am being unable to have the access of the iTunes because the swarm on the MAC is not supporting and it is getting iTunes error 9


#7

@tommyshelby Hi and welcome to HTM Forum! Thanks for your question. I have never seen that error before. Can you tell me the exactly command you ran that cause the error?