We are two students who are trying out some different models for an anomaly detection project.
Our data is timeseries from 40 different sensors, where the values have a close relation to each other, but does not have a close relation to the timestamps. The frequency is one measurement from each sensor each second.
We have tried running the nupic examples on our data without changing too much on the parameters for a singe series, and it seems to spot some anomalies.
The idea we have, is to use the data from all the 40 sensors to spot anomalies in one sensor.
Would HTM be a good choice for this problem or should we go for some LSTM or OCSVM model instead?
If so, how should the data from the 40sensors be encoded to predict only one of the sensor values?
And how memory/compute intensive would such a model be?
If that is the case, there’s no need to encode time as part of the input.
40 fields is way too many for one model. I suggest you keep it to 8 fields max. You might create several models and split the fields between them. Then aggregate the anomaly across all the models?
The only direct comparisons we’ve done between HTM and LSTM have been single scalar field models. I have no idea how LSTM scales up to 40 fields.
In this thread we responded to someone who had a somewhat analogous sounding application with many input fields:
I have a sort of wrapper for NuPIC which makes parallel models for each of the different fields, calibrating each based on the descriptive stats of their first n rows. The anomaly score outputs can then be used to monitor for system anomalies – when a number of the metrics are being anomalous at the same time. How many rows do you have for each of the 40?
We have a continuous incoming stream of data, and also historic data from a year back in time. So we have ~20M rows, and can get more if we need. Our main goal is to detect sensor anomalies, but system anomalies would be useful aswell.
Ok cool! With this approach both are done, since the definition of ‘system anomaly’ is drawn from the current anomaly-levels of the systems’ constituent metrics. It can tell you at any given time step the anomaly score and likelihood of each of the metrics, then some custom logic can be applied to classify each moment as a ‘system anomaly’ – with the list of those metrics which are acting anomalous.
