Results Comparison: HTM Studio vs Nupic

Engineering Applications
1

Hi
Why nupic results are not matching with HTM Studio results ? which one is accurate ?
Pls see the below screenshot of results .

image.png2560×482 78.2 KB

2

You are comparing NuPIC’s “raw anomaly score”, which is not the same as anomaly likelihood. If you want a fair comparison, add AnomalyLikelihood to your NuPIC runtime.

3

Hi Matt ,
Thanks for the sharing the link , it is really helpful ,

if we dont pass any parameter in Anomaly.create() in goes with PURE in both Python and Java

As per below documentation, in PURE mode , nupic calculates raw anomaly score .
MODE_PURE = 'pure’
Default mode. The raw anomaly score as computed by computeRawAnomalyScore()

if that is the case , in above image “anomaly_score” on left side , should match with right side studio raw anomaly_score .

4

Wait, I thought you were comparing the results of HTM Studio and NuPIC? Neither of these are Java-based…

5

Hi Matt (@rhyolight)

Apologies for confusion , I am comparing between Nupic vs HTM studio , screen shot also from the same.

Thanks & Regards
Hareesh

6

It still seems like you are comparing the raw anomaly scores from NuPIC to the anomaly likelihood in HTM Studio. Where is your NuPIC code? I want to see how you are setting up the anomaly / anomaly likelihood class. You should use MODE_LIKELIHOOD when creating the Anomaly class.

7

Hi Matt (@rhyolight)

I am using the below code for anomaly score

Added the below lines to likelihood score after model = createModel() line

  anomalyLikelihood = anomaly_likelihood.AnomalyLikelihood()

in the loop the below piece code to get likelihood

likelihood = anomalyLikelihood.anomalyProbability(
        modelInput["consumption"], anomalyScore, modelInput["timestamp"]
      )
8

@rhyolight Could you pls comments on this .

9

In the chart, it says you’re plotting the raw_anomaly_score from HTM Studio. But you are comparing it to the anomaly likelihood in NuPIC. That still seems like the problem.

10

Hi Matt (@rhyolight)

Apologies for troubling you
anomaly_score is calculated using pure mode(raw_anomaly_score) ,which should equal to “raw_anomaly_score” of studio
image

Thanks & Regards
Hareesh

11

I did not even know that you could get a raw anomaly score out of HTM Studio. You should also be able to get the anomaly likelihood from HTM Studio too, right? More than just a “HIGH” anomaly level, I mean. That is the number you should be comparing to the anomaly likelihood coming out of NuPIC.

Also, I don’t know exactly how models are created and run in HTM Studio, but it is using NuPIC of course. There are probably some small differences in the model configurations between these systems that could cause differences. But overall you should get comparable anomaly likelihood values for the same input data for both systems.