Speed of HTM for near realtime anomaly detection

Community
1

I am looking at this paper “Real-Time Anomaly Detection for Streaming Analytics” (https://arxiv.org/pdf/1607.02480.pdf).

The paper makes it sound like this should be a fast process. I am currently putting ~ 10-20 thousand signals (KPIs) through an anomaly detector per five minutes (looking back only to the top of the current hour and compares that to a model file built offline that looks back one year.)

Does anyone have any experience with something like this and HTM? Know of any large scale projects using HTM for near realtime anomaly detection?

Thank for any help!

2

Depending on how large your network is. 20K/300s (15ms per event) is just around the upper limit of what Etaler can do on a RTX 2080. Tho I’ve never deployed Etaler on large scale projects. Please let me know if you are interested.

3

Thanks for the pointer! I am currently looking to implement as a Splunk add-on (servers with no GPUs).

I am actually looking for a way to grow the number of signals 10x at least … might have to end up offloading from Splunk to other servers with some GPUs.

1 Like
4

Etaler support CPU only mode too. Anyway. 20K is a pretty high for HTM. (NuPIC have no chance reaching that speed). If you’re getting 20K points on a continuous signal, you might want to use traditional statistics. Otherwise a cluster is needed to reach such performance requirement.

1 Like