If the app runs for a long time it crosses the highest memory usage value ~88 % that it has seen in the training and starts showing anomalies (100% anomaly scores). So can’t really comment on periodicity in anomaly graph in that sense.
I had disabled learning during prediction during which I got these graphs as I wanted to see if the periodic pattern has been learnt during training when the learning was enabled.
Ok, I am not an expert on NuPIC, but one possibility is that it has learned a 5-minute long “high memory usage” sequence. When the minicolumns burst, it could be recognizing that it is back at the beginning of the “high memory usage” sequence, and would give a low anomaly score until it again went on for longer than 5 minutes. Just a guess though. More knowledgeable folks may see something else going on.
Interesting, so which time spans is learning enabled for? I’m not sure that a model would pick up too much here, since this data doesn’t look periodic as the prior data was. My guess is the anomaly scores go flat when the memory usage itself is flat, since it has basically learned to just predict a constant value. Then when there’s enough drift from that value given the encoder settings the anomaly score flares up again, before settling down with a new near-constant.
The learning was enabled during training when I created patterns like this :
I disabled learning during predictions when I got this graph:
https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/1/112efcd11d80e4b08bad48cab5afae94de2b1762.png
I expected this pattern to be learnt from training that it has run for more than 5 min since so many similar runs are there in training of 5 min each.
@sheiser1 Is there anything that can be done to make sure that the model learns 5 min pattern and starts flagging anomaly as long as the app runs for more than 5 min ?
I’m not sure how to guarantee that the model shows anomalies continuously after 5 minutes.
If the model has seen a consistent drop in the metric after 5 minutes, then the anomaly score should spike when that expected drop doesn’t occur. However if the behavior after 5 minutes is similar to before, the model will recognize sequences and not be fully surprised.
It does seem possible to supplement with a custom detector for this issue. It seems there’s a sharp spike in memory usage whenever the app starts. You could start a counter at these times, and raise continuous anomalies whenever that counter has passed 5 minutes without a sharp drop (indicating the app has stopped).
