# Outlier prediction using nupic

**URL:** <https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941>\
**Category:** NuPIC\
**Created:** [May 25, 2018, 11:24pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941 "2018-05-25T23:24:44Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [May 25, 2018, 11:24pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/1 "2018-05-25T23:24:44Z")

</div>

I have a time series dataset of around 5k records.There are some outlier values counting 10.My aim is to predict these outlies…For that i tried opf and network models and also did different changes to the parameters like modifying resolution for encoding,alpha value but the outlier prediction is not happening…offcourse it is improving the prediction of commonly seen values.

So I wanted to know if outlier prediction is possible using nupic?If yes…how can i do that…should i change any specific parameter or any codebase which i can use.  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [May 26, 2018, 3:17pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/2 "2018-05-26T15:17:58Z")

</div>

Show me the data! 😉

---

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [May 28, 2018, 1:11pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/3 "2018-05-28T13:11:01Z")

</div>

Here is the data.  
[https://www.pastiebin.com/5b0bff787829a](https://www.pastiebin.com/5b0bff787829a)

Thanks.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [May 29, 2018, 4:45pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/4 "2018-05-29T16:45:23Z")

</div>

@Millan I ran your data through [HTM Studio](https://numenta.com/applications/htm-studio/) after [changing the date format](https://gist.github.com/rhyolight/c7700d14c9075dbe4d1c6d543da3bc68) so it would work. I saw anomalies around Apr 9 in your data.

 ![06%20AM](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/c/cb1d774ab306b32d8a1159b4c289e02c12d77a26.png)

What are the outliers you are expecting it to catch? I don’t see any.

---

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [May 31, 2018, 9:57am UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/5 "2018-05-31T09:57:27Z")

</div>

I am aiming for prediction not anomaly detection.  
In the dataset ,the outliers are the values those are more than or equal to 90 . For these actual values, the prediction values are coming very less .Even for the actual values of more than or equal to 80, the predictions are coming less not impressive.

So basically i want to improve the predictions for higher values.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [May 31, 2018, 12:09pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/6 "2018-05-31T12:09:01Z")

</div>

Do these higher outliers occur with any pattern? Like do they tend to come after certain values or sequences? The system can definitely predict those higher values, but only if they occur as part of some discernible pattern. The more complex the pattern the more repetitions it will take to learn. If they’re basically noise values on the other hand then it won’t predict them and they’ll produce high anomaly scores.

Also it may be worth trying a simple Scalar Encoder with a max-val equal to the minimum out-lier value. This will clip all higher inputs to 90 for example, so the encoder would group anything above 90 into the same encoding (basically a catch-all ‘out-lier’ category). This could help, though only to the extent that the outliers occur with some kind of temporal pattern.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [May 31, 2018, 3:34pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/7 "2018-05-31T15:34:04Z")

</div>

> [@Millan](#):
>
> I am aiming for prediction not anomaly detection.

Prediction is harder. I suggest you [swarm](http://nupic.docs.numenta.org/stable/guides/swarming/index.html) over your data and see what parameters it comes up with if you have not already. How far ahead are you trying to predict?

---

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [June 6, 2018, 11:13am UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/8 "2018-06-06T11:13:50Z")

</div>

Thank you Sheiser1…I ll try out this approach.

---

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [June 6, 2018, 11:16am UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/9 "2018-06-06T11:16:28Z")

</div>

@rhyolight…I have done the swarming. On the swarming result, i did different tuning on the parameter which improved prediction , but for higher values i am yet to see good result.I am trying to predict 4 step ahead.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [June 6, 2018, 3:13pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/10 "2018-06-06T15:13:04Z")

</div>

Thank you for trying. It is hard for the system to predict outliers like without manual resets. For example, if you know the period of the pattern occurring, you could tell the TM to reset its current sequence state, essentially cutting off what it is learning. On the next time step it will start to learn a new sequence. These outliers are parts of a periodic pattern, and the only way to nail it down really is to reset at the same point in the period.

This is a crappy answer, I know, but we honestly don’t really know how temporal patterns are reset in the brain, I talked about this with Jeff in a video.

---

<div class="post-metadata">

**Author:** ![Millan](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@Millan](https://discourse.numenta.org/u/Millan)\
**Post date:** [June 8, 2018, 8:25am UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/11 "2018-06-08T08:25:00Z")

</div>

The higher values mostly occurs somewhere between current day 9pm to next day 4am. So should I reset the sequence at 9pm or at the start of the day(at 0:00 am)?

Currently I am resetting the sequence at start of the day(0:00) which is after 96th records(counter=95) of each day. As I am using algorithm API, below is the code snippet.

```
    tm.compute(activeColumnIndices, learn=True)
    activeCells = tm.getActiveCells()
    if counter % 95 == 0:
      tm.reset()

```

Please suggest if this is the right way to do or should i stop learning as well?

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [June 8, 2018, 2:55pm UTC](https://discourse.numenta.org/t/outlier-prediction-using-nupic/3941/12 "2018-06-08T14:55:14Z")

</div>

That seems right to me. I’m not sure what advice to give you. We have found that HTM in this form provides the most value giving anomaly indications, not predictions.