Hi,
I’m trying to run the code of the hot-gym (hoping to later convert it to learn Cyber related metrics behaviour) and now, after running the code that would create the model I get the following error:
model = ModelFactory.create(modelParams.MODEL_PARAMS)
AttributeError: ‘dict’ object has no attribute ‘MODEL_PARAMS’
Any ideas?
It doesn’t look like the YAML is being loaded properly. I would double check to ensure the file you’re loading actually exists and that the resulting string you are passing to yaml.safe_load() is valid YAML. yaml.safe_load is supposed to return a python object, not a dict. But it looks like modelParams is a dict, so something is wrong before you even create a model.
I sent you the files yesterday to your email. Can you please take a look and tell me what am I doing wrong?
I used this code from the OPF guide:
import yaml
import json
from nupic.frameworks.opf.model_factory import ModelFactory
_PARAMS_PATH = "/home/system/MYTREE/Programming Projects/Python Projects/nupic/model.yaml"
with open(_PARAMS_PATH, "r") as f:
modelParams = yaml.safe_load(f)
model = ModelFactory.create(modelParams.MODEL_PARAMS)
# This tells the model the field to predict.
model.enableInference({'predictedField': 'consumption'})
and downloaded the .yaml file from here:
http://nupic.docs.numenta.org/stable/quick-start/example-model-params.html
Double check to ensure the file you’re loading actually exists and that the resulting string you are passing to yaml.safe_load() is valid YAML. Add some debugging prints to find out:
with open(_PARAMS_PATH, "r") as f:
print f
modelParams = yaml.safe_load(f)
print modelParams
What do you get?
This:
{'spParams': {'columnCount': 2048, 'spVerbosity': 0, 'localAreaDensity': -1.0, 'spatialImp': 'cpp', 'inputWidth': 946, 'synPermInactiveDec': 0.005, 'synPermConnected': 0.1, 'synPermActiveInc': 0.04, 'seed': 1956, 'numActiveColumnsPerInhArea': 40, 'boostStrength': 3.0, 'globalInhibition': 1, 'potentialPct': 0.85}, 'predictAheadTime': None, 'seconds': 0, 'tmParams': {'columnCount': 2048, 'pamLength': 1, 'permanenceInc': 0.1, 'outputType': 'normal', 'initialPerm': 0.21, 'seed': 1960, 'maxSegmentsPerCell': 128, 'temporalImp': 'cpp', 'activationThreshold': 16, 'cellsPerColumn': 32, 'permanenceDec': 0.1, 'minThreshold': 12, 'verbosity': 0, 'maxSynapsesPerSegment': 32, 'globalDecay': 0.0, 'newSynapseCount': 20, 'maxAge': 0, 'inputWidth': 2048}, 'tmEnable': True, 'years': 0, 'hours': 1, 'modelParams': None, 'inferenceType': 'TemporalMultiStep', 'model': 'HTMPrediction', 'sensorParams': {'verbosity': 0, 'encoders': {'timestamp_timeOfDay': {'fieldname': 'timestamp', 'timeOfDay': [21, 1], 'type': 'DateEncoder', 'name': 'timestamp_timeOfDay'}, 'consumption': {'fieldname': 'consumption', 'seed': 1, 'resolution': 0.88, 'name': 'consumption', 'type': 'RandomDistributedScalarEncoder'}, 'timestamp_weekend': {'fieldname': 'timestamp', 'type': 'DateEncoder', 'name': 'timestamp_weekend', 'weekend': 21}}, 'sensorAutoReset': None}, 'trainSPNetOnlyIfRequested': False, 'clParams': {'steps': '1,5', 'maxCategoryCount': 1000, 'implementation': 'cpp', 'alpha': 0.1, 'verbosity': 0, 'regionName': 'SDRClassifierRegion'}, 'fields': [['consumption', 'mean']], 'months': 0, 'days': 0, 'aggregationInfo': None, 'version': 1, 'spEnable': True, 'microseconds': 0, 'weeks': 0, 'minutes': 0, 'milliseconds': 0}
And if I try to “preetify” it with json.dumps and jq I get this:
{
"spParams": {
"columnCount": 2048,
"spVerbosity": 0,
"localAreaDensity": -1,
"spatialImp": "cpp",
"inputWidth": 946,
"synPermInactiveDec": 0.005,
"synPermConnected": 0.1,
"synPermActiveInc": 0.04,
"seed": 1956,
"numActiveColumnsPerInhArea": 40,
"boostStrength": 3,
"globalInhibition": 1,
"potentialPct": 0.85
},
"predictAheadTime": null,
"seconds": 0,
"tmParams": {
"columnCount": 2048,
"pamLength": 1,
"permanenceInc": 0.1,
"outputType": "normal",
"initialPerm": 0.21,
"seed": 1960,
"maxSegmentsPerCell": 128,
"temporalImp": "cpp",
"activationThreshold": 16,
"cellsPerColumn": 32,
"permanenceDec": 0.1,
"minThreshold": 12,
"verbosity": 0,
"maxSynapsesPerSegment": 32,
"globalDecay": 0,
"newSynapseCount": 20,
"maxAge": 0,
"inputWidth": 2048
},
"tmEnable": true,
"years": 0,
"hours": 1,
"modelParams": null,
"inferenceType": "TemporalMultiStep",
"model": "HTMPrediction",
"sensorParams": {
"verbosity": 0,
"encoders": {
"timestamp_timeOfDay": {
"fieldname": "timestamp",
"timeOfDay": [
21,
1
],
"type": "DateEncoder",
"name": "timestamp_timeOfDay"
},
"consumption": {
"fieldname": "consumption",
"seed": 1,
"resolution": 0.88,
"name": "consumption",
"type": "RandomDistributedScalarEncoder"
},
"timestamp_weekend": {
"fieldname": "timestamp",
"type": "DateEncoder",
"name": "timestamp_weekend",
"weekend": 21
}
},
"sensorAutoReset": null
},
"trainSPNetOnlyIfRequested": false,
"clParams": {
"steps": "1,5",
"maxCategoryCount": 1000,
"implementation": "cpp",
"alpha": 0.1,
"verbosity": 0,
"regionName": "SDRClassifierRegion"
},
"fields": [
[
"consumption",
"mean"
]
],
"months": 0,
"days": 0,
"aggregationInfo": null,
"version": 1,
"spEnable": true,
"microseconds": 0,
"weeks": 0,
"minutes": 0,
"milliseconds": 0
}Now the code looks like this:
import yaml
import json
from nupic.frameworks.opf.model_factory import ModelFactory
_PARAMS_PATH = "/home/system/MYTREE/Programming Projects/Python Projects/nupic/model.yaml"
with open(_PARAMS_PATH, "r") as f:
modelParams = yaml.safe_load(f.read())
model = ModelFactory.create(modelParams)
# This tells the model the field to predict.
model.enableInference({'predictedField': 'consumption'})
But this is the result:
Traceback (most recent call last):
File "./createModel", line 12, in <module>
model = ModelFactory.create(modelParams)
File "/usr/local/lib/python2.7/dist-packages/nupic/frameworks/opf/model_factory.py", line 85, in create
return modelClass(**modelConfig['modelParams'])
TypeError: ABCMeta object argument after ** must be a mapping, not NoneTypeYou are still not loading the YAML correctly.
modelParams = yaml.safe_load(f.read())
Not f.read(), just f.
All the examples in the quick start are available in our repository at https://github.com/numenta/nupic/tree/1.0.1/docs/examples and they all run out of the box. For example, from the root of the repository, this works fine for me at NuPIC 1.0.1:
cd $NUPIC
git checkout 1.0.1
cd docs/examples/opf
python complete-opf-example.py
Outputs (partial):
1-step: 39.7232477672 (100.0%) 5-step: 9.78 (100.0%)
1-step: 43.989023697 (94.51%) 5-step: 5.26292042555 (100.0%)
1-step: 39.7232477672 (100.0%) 5-step: 5.26292042555 (100.0%)
1-step: 28.6 (99.98%) 5-step: 5.26292042555 (100.0%)
1-step: 43.989023697 (100.0%) 5-step: 5.26292042555 (100.0%)
1-step: 5.26292042555 (72.24%) 5-step: 17.8 (100.0%)
1-step: 4.76544413354 (100.0%) 5-step: 48.7 (99.75%)
1-step: 5.21404429788 (100.0%) 5-step: 14.127 (100.0%)
1-step: 5.14983100852 (100.0%) 5-step: 4.76544413354 (100.0%)
1-step: 4.76544413354 (100.0%) 5-step: 4.76544413354 (100.0%)
1-step: 19.2 (100.0%) 5-step: 4.76544413354 (100.0%)
1-step: 14.127 (100.0%) 5-step: 4.77581089348 (100.0%)
1-step: 4.77581089348 (100.0%) 5-step: 25.4 (100.0%)
1-step: 4.75306762543 (100.0%) 5-step: 43.989023697 (100.0%)
1-step: 4.7671473378 (100.0%) 5-step: 42.494284 (99.99%)
1-step: 4.7671473378 (100.0%) 5-step: 48.7 (100.0%)
1-step: 25.4 (100.0%) 5-step: 50.15 (100.0%)
1-step: 43.3236159996 (52.05%) 5-step: 37.1137660003 (99.5%)
I would start with these scripts that work just as they are, and change them to your liking.
Hi.
Sorry… I forgot to mention that I tried it without the .read() and it gave the same error so I chose to try it with the .read() after you mentioned that it should read a string…
I tried now to clone your git repo and run the python script you mentioned but under nupic/examples/opf I only see this:
system@Sys-PC:~/MYTREE/Programming Projects/Python Projects/nupic/nupic-git/nupic/examples/opf$ ll
total 24
drwxrwxr-x 6 system system 4096 אוג 20 00:28 ./
drwxrwxr-x 10 system system 4096 אוג 20 00:28 …/
drwxrwxr-x 5 system system 4096 אוג 20 00:28 clients/
drwxrwxr-x 10 system system 4096 אוג 20 00:28 experiments/
drwxrwxr-x 2 system system 4096 אוג 20 00:28 simple_server/
drwxrwxr-x 3 system system 4096 אוג 20 00:28 tools/
This is the git clone I used:
git clone https://github.com/numenta/nupic.git
Well - WOW… Thanks a lot… I tried all your suggestions at 01:00 AM my time so no wonder I wasn’t focused enough but now I even managed to add the anomaly score to the example you pointed me to and that was great!
Now I think that I just need a little more help:
I one of your videos you show that you call something from nupic_output to plot the results on a chart. I tried to import it or to find it in the git anywhere… Is it still possible to do it? It was so cool when you plotted it like that!
Also, if I have another data that I would like to try to run through this anomaly detection mechanism, it is also a data that should behave similar to the hotgym: it should decrease on weekends and during night-time. Is there something that I should do except for simply replacing the CSV data file?
Thanks a lot for all your help,
Yuval Khalifa
See https://github.com/numenta/nupic/tree/master/examples/opf/clients/hotgym/prediction/one_gym and the video for explanation. You can find nupic_output here.
You should be okay just changing out the values. But if you change the input format or the field names, you’ll also need to update the model parameters.
Hi
I’m sorry to bother you again with a (probably stupid) question. Here it is:
Thanks for your help I managed to run the hot-gym example with plotting by using this script which is a modified (by me) version of the original complete-opf-example but I don’t know how to make the plot include the anomaly score (like you do on the sine wave example):
import csv
import datetime
import os
import yaml
from itertools import islice
import nupic_output
from nupic.frameworks.opf.model_factory import ModelFactory
_NUM_RECORDS = 3000
_EXAMPLE_DIR = os.path.dirname(os.path.abspath(__file__))
_INPUT_FILE_PATH = os.path.join(_EXAMPLE_DIR, os.pardir, "data", "gymdata.csv")
_PARAMS_PATH = os.path.join(_EXAMPLE_DIR, os.pardir, "params", "model.yaml")
def createModel():
with open(_PARAMS_PATH, "r") as f:
modelParams = yaml.safe_load(f)
return ModelFactory.create(modelParams)
def runHotgym(numRecords):
output = nupic_output.NuPICPlotOutput(["hot-gym"], showAnomalyScore=True)
model = createModel()
model.enableInference({"predictedField": "consumption"})
with open(_INPUT_FILE_PATH) as fin:
reader = csv.reader(fin)
headers = reader.next()
reader.next()
reader.next()
i = 0
results = []
for record in islice(reader, numRecords):
modelInput = dict(zip(headers, record))
modelInput["consumption"] = float(modelInput["consumption"])
modelInput["timestamp"] = datetime.datetime.strptime(
modelInput["timestamp"], "%m/%d/%y %H:%M")
result = model.run(modelInput)
bestPredictions = result.inferences["multiStepBestPredictions"]
allPredictions = result.inferences["multiStepPredictions"]
anomalyScore = result.inferences["anomalyScore"]
oneStep = bestPredictions[1]
oneStepConfidence = allPredictions[1][oneStep]
fiveStep = bestPredictions[5]
fiveStepConfidence = allPredictions[5][fiveStep]
result = (oneStep, oneStepConfidence * 100,
fiveStep, fiveStepConfidence * 100, anomalyScore)
output.write([modelInput["timestamp"]], [modelInput["consumption"]], [bestPredictions[1]], [anomalyScore])
print "1-step: {:16} ({:4.4}%)\t 5-step: {:16} ({:4.4}%)\t anomalyScore: {:16}".format(*result)
results.append(result)
if i % 10 == 0:
output.refreshGUI()
i = i + 1
return results
if __name__ == "__main__":
runHotgym(_NUM_RECORDS)
Any ideas?
See the hot gym anomaly example code, specifically nupic_anomaly_output.py.
DISCLAIMER: All this plotting code is matplotlib, which I hate and I’m not very good at. Use at your own risk. I cannot really provide much support for plotting libraries.
Thanks a lot. I just tried it and I get a funny result… I attached a screenshot to make it easier to explain. I downloaded the exact files from the example code you pointed me to and tried to run it with (and without) the --plot and when I use --plot I see the chart window but it’s empty…
When I used the previous OPF example it worked just fine and plotted both the predicted and the consumption very well…
What am I missing?
P.S
As I mentioned earlier I’m only turning to you so quickly without googling around first because I’m really under a lot of pressure to demonstrate this framework next week so I must make it work REALLY quickly…
Tnx,
Yuval Khalifa
Blog. http://www.artifex.co.il/en
I’m sorry, but I can’t help you with your matplotlib issues. It is a tricky platform that doesn’t always work out of the box in every environment. This might point you in the right direction (I found that link doing a forum search for matplotlib).
You could always just write the output to a file and plot it afterwards.
I think you’ll get better and faster results from googling them yourself than posting questions here and waiting for answers. Then, when you do resort to posting questions here, they will be better informed and more likely to get a good answer.
