Int Assertion Error

Use TemporalAnomaly. This InferenceType is standard because it tells the cla model factory to use the temporal pooler (i.e. temporal memory); I found some documentation hidden here, with the caller here.

Hello,

I think the given solution of “getScalarMetricWithTimeOfDayAnomalyParams” is not working for me, as it also wants a specific input field for “predictedField” in params.

Maybe if you shared your code we could help more.

Here is run.py

Here is model param

If I don’t give predictedField in model params or disable model.enableInference(params[“inferenceArgs”]), I get following error:

ValueError: No predicted field was enabled! Did you call enableInference()?

Even though you have imported this function on line 10:

from nupic.frameworks.opf.common_models.cluster_params import (
        getScalarMetricWithTimeOfDayAnomalyParams)

You are not using it anywhere. Replace this:

model = createModel(getModelParamsFromName(app))

with something like this:

params = getScalarMetricWithTimeOfDayAnomalyParams(
  metricData=[0], # just dummy data unless you want to send in some real data here
  minVal=<your min data value in your CSV file>,
  maxVal=<your max data value in your CSV file>,
  minResolution=0.001 # you may need to tune this
)
# Here, you can print out the params cause its is just a dict, and change
# them to suit your needs. Here, I'll just print them out so you can see them:
from pprint import pprint; pprint(params)
# Now use these params to create a model
model = createModel(params)

@rhyolight I have created model_params using getScalarMetricWithTimeOfDayAnomalyParams
As you had suggested, I called that function 3 times, since I wanted to get params for my 3 input fields.

Pls look at the default model params that this function returns. It has “inferenceArgs” within which, it has “predictedField”

@rhyolight
Following is my code to get model params for 3 fields. After getting them, I had constructed the model param manually,

Following is my code, the main input argument were min and max value:

import sys
import pprint

from nupic.frameworks.opf.common_models.cluster_params import (
        getScalarMetricWithTimeOfDayAnomalyParams)


def getParams(columnNb,min,max):

    params = getScalarMetricWithTimeOfDayAnomalyParams(
        metricData=[0],
        minVal=float(min),
        maxVal=float(max))
    
    pprint.pprint (params)


if __name__ == '__main__':
    inputParam=sys.argv[1:]
    print inputParam
    columnNb=[0]
    min=inputParam[1]
    max=inputParam[2]
    getParams(columnNb,min,max)

You are close.

The first time you call getScalarMetricWithTimeOfDayAnomalyParams, use that as your primary model params object. Then for the 2nd and 3rd calls, just look into the encoder parameters and extract the ones that match the input data for each call, rename them, and add them to the list of encoders you have in the primary model params object. Then pass just that one object into createModel().

@rhyolight Yeah, I have done that only. After doing all that you have suggested, I still have to provide “predictedFileld” value.
Did you got a chance to look at the primary params object getScalarMetricWithTimeOfDayAnomalyParams returns?
https://github.com/numenta/nupic/blob/master/src/nupic/frameworks/opf/common_models/anomaly_params_random_encoder/best_single_metric_anomaly_params_cpp.json,
It has “predictedFileld” in it.

If you are only worried about anomalies and not prediction, I don’t think it matter which field is specified as the predictedField. You just need to choose one.

@rhyolight Thank you, that answered my worry.

Would model param for a string field would be ‘’: None ?
The string represents a category. Do you suggest to convert String to Int and then use CategoryEncoder?

Wouldn’t it be flag, packets, bytes, or duration based upon the data sample you posted above?

Yeah, it is flag column. But its values are in String like “FSPA”, “PA” etc…
My combination of packets, bytes, and duration (All of numerical field) is not able to detect anomaly.

It could be that there is no pattern in the input data to recognize. It depends on the data. What do your anomaly scores look like when plotted along with the input data?

Yes, even I am thinking my data is not suitable for NuPIC.
Anomaly score starts with 0.5 and then goes to 0. It remains 0 even while supplying anomaly dataset.

I just want to check one more time by including flags. If is is not working then I will try some different algorithm.

Remember that there is a difference between “anomaly score”, which is the raw score coming out of NuPIC, and “anomaly likelihood”, which is a post-process applying a type of smoothing over a sliding window to the raw anomaly score. The anomaly likelihood seems to be more valuable. This metric usually starts at 0.5 and stays there for 500 rows of input data (the size of its sliding window) before it begins to emit useful values.

If you are talking about the anomaly score, is seems abnormal that it would start at 0.5, drop to 0, and then remain there. I’ve never seen that unless there is some problem with the input data (perhaps you are sending scalar values in as strings? be sure to cast strings to numbers first).

@rhyolight I am only passing numerical data (packets, bytes and duration)

I am not calculating AnomalyLikelihood as I don’t have proper time-stamp field (its in sub seconds)

anomalyLikelihood = anomaly_likelihood.AnomalyLikelihood()
likelihood = anomalyLikelihood.anomalyProbability(
consumption, anomalyScore, timestamp
)

How should I calculate AnomalyLikelihood in such case? Should I truncate seconds and minute parts, but in that case I will get lot of data in the same time?

@Komal I would love to continue helping you, but I am really busy the rest of this week, and I think the best way to help would be for you to get your code into a source control system so you can easily share it. That way we can work on it together and I can point out specific changes that need to be made. Also, even a short data sample would be helpful. If you can do this, I can come back next week (or perhaps this Friday) to help you more.

Ok, no problem. I will wait until you are free. I have already posted the code. Where can I share data sample in private?

Yes, but I’m sure you’ve changed it since you posted it. If it is in source control, it is much easier to collaborate on it. If you can’t, I understand, but I strongly recommend it.

You can send me a private message by clicking on my avatar and then “Message”.