Question about htm.core anomaly detection

Engineering NuPIC Community Fork
, ,
1

Preface: Since I’m a new user I’m not allowed to post multiple images. They are all in this gallery, and I will say the name of the image when it should be referenced in the gallery. Album — Postimages

Hello, I’m new to working with HTM. I’m trying to use it for anomaly detection and have run into a problem. I’m using htm.core and the tm.anomaly function isn’t triggering an anomaly when it should be. I’m using a slightly modified version of the hotgym.py example. My results from this dataset seem to match what I’ve seen from other people running the code.

I believe my confusion mostly lies around the anomaly likelihood class. I believe this is pretty critical to what I want to do based on reading other forum posts and watching matts video on hotgym with NuPIC (About Anomaly Detection Thresholding). Am I correct in thinking that the anomaly likelihood class in htm.core is not functioning properly? I think I read this somewhere but I am not sure. I may be using it wrong or have poor parameters.

Here is me running hotgym, using the same parameters as the original, but also plotting anomaly likelihood:
orig_params.png

Here is a zoomed in version of the image above. It shows correct prediction, but fails to detect the anomaly where power usage doesn’t go back to the same base level at ~320:
orig_params_zoom.png

At first I wondered if the weird weekly spikes was throwing it off. I found increasing the encoder size for the timeOfDay parameter from 30 to 100 removed this anomaly spiking. I assume it just wasn’t large enough for the temporal memory algorithm to remember it well enough with only 30, but I really don’t know why this worked. Here is the two same images from before with new param to show that prediction still works, but anomaly stays glued to 0 and anomaly likelihood stays glued to 1:
new_param.png
new_param_zoom.png

I also tried this against a dataset from NAB with an artificial anomaly and did not catch it:
new_param_vs_anomaly.png

Is my problem just that htm.core doesn’t fully implement anomaly detection. If so I can change to NuPIC, I just didn’t want to deal with python 2 mostly. In case its poor parameters, here is my full parameter list. My htm code is the same as the example from htm.core/py/htm/examples/hotgym.py. My full parameter list is

enc:
  time:
    timeOfDayVar1: 100
    timeOfDayVar2: 1
    weekend: 21
  value:
    resolution: 0.88
    size: 1000
    sparsity: 0.02
predictor:
  sdrc_alpha: 0.1
anomaly:
  likelihood:
    probationaryPct: 0.1
    reestimationPeriod: 100
sp:
  boostStrength: 3.0
  columnCount: 1638
  globalInhibition: True
  localAreaDensity: 0.04395604395604396
  potentialPct: 0.85
  synPermActiveInc: 0.04
  synPermConnected: 0.13999999999999999
  synPermInactiveDec: 0.006
  wrapAround: True
tm:
  activationThreshold: 17
  cellsPerColumn: 13
  initialPerm: 0.21
  maxSegmentsPerCell: 128
  maxSynapsesPerSegment: 64
  minThreshold: 10
  newSynapseCount: 32
  permanenceDec: 0.1
  permanenceInc: 0.1 
  predictedSegmentDecrement: 0.0
2 Likes
2

Hey @zrosin, welcome!

I’m not an htm.core expert, though given the nature of your issue I suspect it’s an issue with either data granularity or encoding – and I’d suggest focusing on anomaly scores (not anomaly likelihoods) to resolve it.

To me this illustrates the problem:

Screen Shot 2021-06-17 at 10.27.43 AM
Screen Shot 2021-06-17 at 10.27.43 AM1288×974 162 KB

That flat area around tilmestep 3000 should show some spiking in anomaly scores.

I’d recommend reducing the data granularity – like aggregate it down so the same basic pattern is shown but over many fewer time steps. This will make it much easier for the TM to learn quickly. The anomaly scores should be high and volatile at first, and then settle to near 0 after several repeats of the pattern.

In your case however it seems the anomaly scores settle down after only ~1 repeat of the larger pattern – which suggests they’ve likely learned shorter term patterns. The TM will eventually learn longer term patterns too – but the longer the pattern the more repetitions it takes to learn. Basically I think your current model is losing the forest for the trees.

So I’d try and aggregate so that the larger pattern only takes say 10 time steps – then show the model 10 repeats of that pattern and check how the raw anomaly scores behave. It should be high & volatile at first and start the settles after several repeats of the pattern.

I’d also add in periods of noise which happen after the anomaly scores have settled – maybe one after 5 repeats and another after 10. This is to make sure that those anomaly scores which have settled to the pattern spike up when the pattern recedes/disappears.

To be thorough you could add different types of noise – like maybe 1 flat line and another with random values in the same min/max range as the pattern. I’ve also done a backwards version of the pattern – which should be anomalous at first but also be learned after some repeats if TM learning is on.

If this aggregation doesn’t get the anomaly scores acting right I’d then look at the encoders – to make sure the proper amount of overlap is there between similar & dissimilar inputs. For this debugging purpose I think it’s easier to work with the Standard Scalar encoder (rather than RDSE) – but either way the point is to ensure that similar values have high overlap in their encoding vectors and dissimilar values have low/no overlap.

Welcome again to HTM and I hope this helps!

2 Likes
3

Thanks for the help! I’ll see how this works.

1 Like
4

I’ve made a new dataset and it appears that shortening the periodicity of the data improved the performance a lot.
better_results

I reverted my parameters back to the hotgym’s original set and it seems to do a lot better with the shorter period, however it still doesnt work against the long period anomaly I have in my original post. Is the culprit there likely some other bad parameter or a bad choice of encoding like you mentioned?

1 Like
5

Alright cool!

Looking at this plot I’m wondering why the anomaly scores:

  • drop instead of jump at the spike around tilmestep 300

  • drop instead of jump at the shift up just after tilmestep 400

  • creep back up around tilmestep 600 after no change in the data at that time

This does make me highly suspect it’s an encoding issue. I’d try dropping any time of day/timestamp encoders (just use the raw numeric input alone). These could be hindering the encodings.

For the toy data I assume you don’t have any timestamp feature, so I’d try swapping the RDSE for a Scalar Encoder – where you set a min & max instead of resolution. For the toy data you could just use the min & max, since there are no extreme outliers to warp it.

Also I’d inject a bit of noise into the data rather than having these perfect sine waves.

I’m curious to see another plot like this with those tweaks.

6

Thanks for the help. I’ve added this as an issue on the htm.core repository because the behavior is so strange.

I’ve added your suggestions and I’m not really sure what to make of the plots. The number encoder didn’t really have an effect but removing the date encoder somewhat improved the anomaly score, though it also caused to to be hypersensitive and react to any change it saw.

The test against noise was poor as well. I gave it a noisy sine wave, and with and without the date encoder it was never able to settle down.

I’ve got a lot of plots so they are all here: Album — Postimages
This is just an explanation in case the filename is not rich enough

  • noisy_anom.png - Fails to detect anomaly, never understands sine wave pattern
  • noisy_date.png - Noisy sine waves
  • noisy_no_date.png - remove date encoder, no real change

I’ve run the model against this noisy data with 10000 data points and it never settles down. It’s possible its just too much noise though.

  • no_date_custom.png - Shows spatial anomalies maybe detected?
  • no_date_flatmiddle.png - Same as above, note that anomaly score is definitely dropping instead of rising though.
  • rdse_custom.png, scalar_custom.png - shows that the number encoding method is giving mostly same results between these.
1 Like
7

Nice, I’m curious what comes of that.

Yea me neither, except that something seems wrong with the way the models are learning. If you feed in a sine wave (or any perfectly repeating pattern) the anomaly score spike should become fewer & further between as the model learns the higher order transitions – those that repeat less frequently. I wouldn’t do anything else before getting the anomaly scores to behave like this for a simple sine wave.

You used the default sp & tm param settings, right? If that’s not the problem I suspect its in the encoding settings.

You did compare the Scalar & RDSE encoders right, and they were similar? Did you use 0 & 100 for the Scalar encoder min/max values?

I’d say reduce the length of the sine waves – the number of time steps it takes to repeat – and drop the noise. Maybe a sine wave with sequence length=5 repeated 50 times, using a Scalar with min/max = 0/100. If the anomaly score spikes haven’t become quite sparse by the end there’s definitely a problem.

If this problem persists I’d dig into the encoder and make sure that the proper amount of bits are overlapping between different inputs. For instance with min/max = 0/100, inputs 1 & 2 should be near totally overlapped. Inputs 1 & 5 should be largely overlapped and inputs 1 & 30 shouldn’t overlap at all.

Another thing that could be informative is the level of prediction activity – number of predicted cells at each time step. I’m not sure how that info is stored in htm.core, but it should be accessible from the TM object. The number of predictions for a perfectly repeating sequence should start at 0, then spike up over 1 (how far over 1 depends on the level of ambiguity in the sequence), and then settle down to 1 eventually as the model learns to predict more precisely.