Htm.core vs NuPIC on NAB

[sheiser1]

Hey all,

So I’m trying to debug the difference in NAB score between NuPIC (as used by the NumentaTM detector) and htm.core.

In my implementation I have aligned all the model hyperparams (SP, TM & Encoders), and also used the same AnomalyLikelihood function.

In spite of this I’m still getting NAB score around 50 for htm.core, while NumentaTM is up around 65.

As I understand NumentaTM detector doesn’t using BacktrackingTM (with tmImplemetation=‘tm_cpp’), so the anomaly scores should align - though that’s not what I’m seeing.

I assumed it was due to some functionality difference in htm.core, but then saw here https://github.com/htm-community/NAB/pull/15 that @breznak and @Zbysekz you guys scored over 60 on NAB with htm.core?? If possible I’d love to check it out and find where my issue is.

Thanks again

[sheiser1]

Ok I got the NAB score up to like 65 by adding the spatial anomaly check (with spatial_tolerance=0.05) used in the other HTM implementations!

[thanh-binh.to]

@sheiser1 congratulation.
Could you please write more detailed how you run ans setup it with htm.core so that any newcomer can learn and test it quickly?
Thanks

[David_Keeney]

Hi guys,
I did some code cleanup work on some of the C++ code in htm.core but the NAB implementation was mostly @ctrl-z-9000-times and @breznak. So I cannot help you much.

[hrodrigon]

@sheiser1

adding the spatial anomaly check

Hi, I saw this topic and I am also trying to obtain a score similar to numentaTM. In which file or script should I try adding the spatial anomaly check?

Thanks in advance,

Helena

[sheiser1]

@hrodrigon

In which file or script should I try adding the spatial anomaly check?

It should be where you calculate the anomaly likelihood. Normally the anomaly likelihood is only based on raw anomaly scores, but in cases of ‘spatial anomaly’ the likelihood is set to 1.0. So after you calculate anomaly likelihood based on anomaly score, have an

if spatial_anomaly: 
anomaly_likelihood=1.0

Here’s how the NumentaTM does it in NAB:

Hope this helps!

[hrodrigon]

Hi! I have reviewed the code and I think that it is already considering the spatial anomaly: anomalyScore = max(spatialAnomaly, temporalAnomaly). However, I have also trying adding the conditional, but it gives a similar result (NAB standard score around 50).

I am trying to figure out if there is another reason that makes me obtain a htmcore NAB score around 50. For the htmcore_detector I have the parameters hardcoded, should I use the same that it uses for numentaTM or htmjava? How could I make it?

Thanks again.

[sheiser1]

@hrodrigon

should I use the same that it uses for numentaTM or htmjava? How could I make it?

Yes I’d say so.
I’d be kinda surprised if the hyperparams alone raises your NAB score that much, but here are the params the numentaTM detector uses.

numenta/nupic-legacy/blob/master/src/nupic/frameworks/opf/common_models/anomaly_params_random_encoder/best_single_metric_anomaly_params_cpp.json

[dmac]

Sigh, I may have screwed this by rewriting the anomaly likelihood code…
The original code from numenta was a hot mess full of heuristics and hardcoded thresholds so I replaced it with a simple and understandable method which is both well grounded in statistical theory and in numenta’s notes on the topic (but not ofc their source code).

[sheiser1]

@dmac

I replaced it with a simple and understandable method which is both well grounded in statistical theory and in numenta’s notes on the topic

Sounds good! Have you compared it with NuPIC’s AnomalyLikelihood results? To replace NAB results using htm.core I had to use NuPIC’s AnomalyLikelihood since the htm.core one was giving crap results, which is a hot mess yes. If there’s a cleaner version of AnomalyLikelihood that achieves comparable I’d gladly try it instead of the mess.