In the process of improving NAB by testing anomaly detection algorithms for addition to the repo, @lscheinkman has opened a PR that implements the HTM.java detector. Matching the parameters to the current HTM detector, we would expect very similar results, but are finding that the HTM.java detector’s scores on NAB are on par with making random detections.
@cogmission has done a careful job of ensuring 100% test viability between the HTM.java and NuPIC implementations, implying there is a bug
- somewhere not covered by the tests
- on the use of HTM Java in NAB
- or some implementation differences between java and python that would only manifest in certain situations
Would anyone in the community be able to help us out and investigate this issue? An initial step could be comparing hotgym results using both NuPIC and HTM.java, moving on to NAB data files later.