Inability to accumulate minor anomalies?

Engineering NuPIC
,
1

Dear all,

this is a question regarding the limitations of HTM as well as the optimal configuration in a certain setting, or at least a suggestion for what could give good results.

We are tracking money flow in a simulated bank and introduce anomalies (large payments) to see if HTM detects it.

Our HTM model is trained on approximately 378.000 data points before learning is disabled. The data has a clear pattern to it, but there is also some noise. The “clear pattern” is a series of large payments (e.g. invoices, large purchases), whereas the noise is simulating smaller more random payments such as groceries. Usually HTM would see inputs ranging from a couple hundred, to at most 25 000. Due to HTM’s problem with sub-second precision, the data points are always more than one second apart from each other.

HTM does reliably detect extremely large payments. If there is suddenly a payment of 1 billion, HTM detects it with an anomaly score of 1.0. However, HTM does not reliably detect 1000 instances of 10 000 being transferred (HTM score is around 0.3).

The question is, why not? Why is it not abnormal to see a long stream of 10 000? Is it because HTM considers 10 000 to be perfectly normal, since it has learned that inputs of size a couple hundred up to 25 000 is normal?

The other question is, what can be done about this?

We are using TemporalAnomaly model with the recommended single metric anomaly parameters (https://github.com/numenta/nupic/blob/master/src/nupic/frameworks/opf/common_models/anomaly_params_random_encoder/best_single_metric_anomaly_params_tm_cpp.json). The following encoders are used:

'encoders': {
   u'money': {
       'fieldname': u'money',
       'name': u'money',
       'resolution': 1000,
       'seed': 42,
       'type': 'RandomDistributedScalarEncoder',
   },

   'timestamp_timeOfDay': {
       'fieldname': 'timestamp',
       'name': 'timestamp',
       'timeOfDay': ( 21, 9.49),
       'type': 'DateEncoder'
   }
}

Any guidance or suggestions are welcome.

The following topic was suggested as relevant, but after having looked at it I don’t think that it is: Not Getting Anomaly Results

1 Like
2

It is not a problem, really, just that the system will see those transactions as happening at the same point in time, which should be accurate enough if you are looking for anomalies at a larger time scale.

The “HTM score”… what is that? Are you talking about the “anomaly score”? If so, you should certainly try using the anomaly likelihood process as well. Here is a post where I tried to explain how to use it:

3

The long stream of 10,000 may not be seen as anomalous because the system learned the new pattern quickly and was therefore no longer surprised by it. If its really a totally new pattern the system should see it as anomalous at least at first before adjusting to the ‘new normal’. I wonder if it may not be seen as anomalous even at first because the jump up to 10,000 (let’s say from some more common value like 200) may have some semantic overlap with other jumps that had occurred. Maybe this was the first time it went form 200 to 10,000, but if it had gone from say 150 to 8000 or something that jump would have similarity, so the new jump from 200 to 10,000 would be novel, though not completely due to the overlap from prior jumps.

4

Thanks for the suggestion. However, it appears that the advice of using the anomaly likelihood instead of the raw anomaly score was given to a user who experienced too many anomalies, i.e. false positives. We, however, have too few anomalies while performing an attack, i.e. false negatives. By attack we mean introducing abnormal transactions.

We’ve tried to use anomaly likelihood as suggested, but it appears to make no significant difference.

Sorry for not making the initial post more clear. By “HTM score” we do indeed mean the raw anomaly score. Also note that the normal range is from 300 to 25.000.

Expected results:
Performing one thousand transactions at 10.000 each should at some point start producing an anomaly score of 1.0. We expect to see the anomaly starting at around 0 and then building up to 1.0 as the continuous stream of 10.000 keeps being observed. Note that seeing a single 10.000 transaction by itself is not an anomaly, even for a human observing the input. However, seeing a long enough stream of these 10.000 should obviously result in a an anomaly.

Actual results:
We find that the anomaly score rarely jumps up to 1.0 (while performing the abnormal 10.000 transactions) but stays for the wast majority of the time close to 0.3. It appears that HTM has no memory of previous events. E.g. it doesn’t matter whether or not we do 10 transactions or 2000 transactions (of 10.000 each).

Are our expected results reasonable? Should we expect to see something else?

Is it even possible for HTM to accumulate abnormal results?

By accumulate we mean accumulating a number of abnormal values, such as 0.1, 0.1, 0.1, …, which should eventually lead to a full anomaly of 1.0. Can HTM do this, either with learning on or off?

5

I would love to see your data. Even if you cannot share it directly, can you show a chart of the data over time? It might help us to understand why this is happening.

6

I wonder if there may be some confusion about the meaning of anomaly score, so I just want to go into it a little in case it helps. Raw anomaly score is basically a measure of how surprised the system is by a given input at a given time step, not cumulatively over multiple time steps.

When an input is perfectly predicted its anomaly score will be 0 and when its completely caught off guard it will be 1.0. If one occurrence of 10,000 is not too unusual but a long string of 10,000’s is, I’d expect the system to be surprised when 10,000 comes in a second, third and fourth time in a row. Once it keeps coming in like this though, a new pattern of these repeated 10,000’s is established and NuPIC will start to predict them (and thus not be surprised to see 10,000 anymore). Once the novelty of the new pattern wears off the anomaly score will drop back down.

I bring this up because I don’t think that the anomaly score itself will “build up to 1.0” but rather spike up to 1.0 when the new pattern first shows up, then drop back down once it gets used to it. I’d be curious about those points when it jumps to 1.0. That implies that its totally surprised by the input at that time.

Also the ‘anomaly likelihood’ may still be useful. It is a normalization of the raw anomaly scores over time that basically tells you when the predictability of the patterns changes. In noisy streams NuPIC will produce a lot of high anomaly scores, since it doesn’t have a reliable pattern to make predictions on and is surprised a lot. If you just look at the raw anomaly score you could therefor wind up with a lot of false positives, so the likelihood is there to compare the current anomaly score at each time step to the average score over the past x-number of time steps (I think 300 or so but I don’t remember). If there have been a lot of high anomaly scores then one high anomaly score isn’t really surprising, but a string of low anomaly scores would be. Likewise the other way around, if the anomaly scores were mostly low and then several came in high in close succession that would have a high anomaly score. I say all this because I the anomaly likelihood is the established way of normalizing anomaly score values over time period longer than one time step.

– Sam

7

Thanks for the help. We have attached two plots.

The first plot represents the input given to HTM during an attack. As you can see it has a very clear pattern. The data observed at 12 000 (Y axis) is the “attack data”, which has never been seen before at all in the previous data given to HTM. We expected that HTM would report this continuously as an anomaly with learning off – since it has never seen it before. We especially expected it to register it as a continuous anomaly when observed repeatedly, at least until it has learned the behavior. With learning off, we should expect to see a continuous 1.0 raw anomaly score.

input_data.png1024×768 7.39 KB

The second plot shows the anomaly score for a full data set where the attack is only a minor part of it, at the very end. There is no anomaly detection observed at all in this attack region. This causes a large number of observations to be grouped at the same year (i.e. the exact coordinate for that year). For the same reason there will appear to be multiple anomaly scores for the same X value, these are however distinct (or groups of) anomaly scores for discrete observations on the X axis.

output_data.png1024×768 7.32 KB

Can it be that the used HTM parameters are incorrect? Can there be anything else obviously wrong with our setup?

8

This is not actually true, because there are still normal data involved, it is not 100% anomalous. Just because it gets input that its never seen before doesn’t mean the anomaly score jumps to 1.0.

After looking at the chart of the input data, I agree that HTM should be indicating anomalies for the continuous 12,000 values. But I’m not certain that everything is programmed properly. There are a lot of things that could be wrong. Can you share the code? Are you certain that you are casting string input into scalar values before passing to the model?

I also think you should plot the anomaly likelihood values, they are generally much more useful than anomaly score.

9

What do you mean by scalar values? The input values are constructed in the following manner:

model.run({
    "timestamp": datetime.strptime(timeStr, TIME_FORMAT),
    "money": float(value)
})
10

I meant float(value). If you send in a string, it gets cast into a number. You’re doing it right.