How to get Anomaly Likelihood using anomaly_likelihood.updateAnomalyLikelihoods

Engineering NuPIC
1

Looking for some quick guidance on running the online version of the anomaly likelihood, using the ‘anomaly_likelihood.updateAnomalyLikelihoods’ function. In the source (anomaly_likelihood.py) it says:

Initially:
… code-block:: python
likelihoods, avgRecordList, estimatorParams = \
estimateAnomalyLikelihoods(metric_data)
Whenever you get new data:
… code-block:: python
likelihoods, avgRecordList, estimatorParams = \
updateAnomalyLikelihoods(data2, estimatorParams)

My issue is that since the model hasn’t seen any data yet there is no history of anomaly scores to pass into updateAnomalyLikelihoods(). Here’s what I’m doing now:

dist = {'mean':0.5, 'name':'normal', 'variance':1e6, 'stdev':1e3}
estimatorParams = {
    "distribution": dist,
    "movingAverage": {
        "historicalValues": [],
        "total": 0,
        "windowSize": 1,
    },
    "historicalLikelihoods": [],
}

likelihoods, avgRecordList, estimatorParams = anomaly_likelihood.estimateAnomalyLikelihoods(anomalyScores=[ [0,0,1.0] ],averagingWindow=1)

I know the ‘anomalyScores’ argument to estimateAnomalyLikelihoods() consists of a timestamp, raw data value and raw anomaly score. Though since my data does not have a timestamp column I pass in the index value of 0, along with another 0 for the first raw data point and 1.0 for the first raw anomaly score.

Once the initial estimateAnomalyLikelihoods() has run I iterate over the data frame rows, with each new data point running:

AnomScore = resultObj.inferences["anomalyScore"]
likelihoods, avgRecordList, estimatorParams = anomaly_likelihood.updateAnomalyLikelihoods([ [index,row[field],AnomScore] ], estimatorParams)

if index%AnomLikl_obj.reestimationPeriod == 0:
    AScores = [ [v['index'], row[field],v['AScore']] for v in results ]. ##gather all raw anomaly     score so far##
    likelihoods, avgRecordList, estimatorParams = anomaly_likelihood.estimateAnomalyLikelihoods(
        anomalyScores=AScores[-AnomLikl_obj.historicWindowSize:], averagingWindow=1)

Notice anything wrong looking or missing in there? As of now it will run without error and output raw anomaly scores but no likelihoods. Thanks

2

Sam are you still having this issue? I may investigate it tomorrow during my live-stream.

1 Like
3

I have not gotten past this yet no, so if you’d be willing to take a peak thats great! If I understand right, using the helper class

AnomLikl_obj = anomaly_likelihood.AnomalyLikelihood()

results in the Anomaly Likelihood value at each time step being the moving average over the last 10 points.

def estimateAnomalyLikelihoods(anomalyScores,
                               averagingWindow=10,
                               skipRecords=0,
                               verbosity=0):

I want to control that ‘averagingWindow’, though something is wrong in my use of that function. It seems like setting the ‘averagingWindow’ to 1 would makes the system most quickly adaptive from one time step to the next. I’m not sure if it matters much in practice, though still want to have the proficiency to control it.

1 Like
4

Let me point you to an example that might help.

This creates a helper, and then I called anomalyProbability() on it:

This looks a bit different than your code. Not sure if this helps.

Sorry I went way off topic when answering.

1 Like