# Can HTM be used on multivarient time series problems?

**URL:** <https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316>\
**Category:** Engineering\
**Created:** [January 17, 2018, 7:09pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316 "2018-01-17T19:09:10Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 17, 2020, 9:09am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/22 "2020-04-17T09:09:22Z")

</div>

Hi @steinroe.  
I actually implemented a Multivariate Anomaly Detection (with 1 single model), and the results are pretty encouraging. The code is still messy (need to do some refactoring). Here is what I did: [https://buttignon.next-cloud.org/index.php/s/D9CJmGMoADB56rd](https://buttignon.next-cloud.org/index.php/s/D9CJmGMoADB56rd)

---

<div class="post-metadata">

**Author:** ![steinroe](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@steinroe](https://discourse.numenta.org/u/steinroe)\
**Post date:** [April 17, 2020, 9:23am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/23 "2020-04-17T09:23:21Z")

</div>

Hi @pizzatakeaway,

thanks for sharing!! Would it be possible to share to whole document? I would like to cite it in my work 🙂

Best  
Philipp

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 19, 2020, 8:06pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/24 "2020-04-19T20:06:52Z")

</div>

Hey @steinroe, so I have my own python module which builds multiple NuPIC anomaly detection models from multivariate time series.

It samples the first N rows to calibrate the encoders, then instantiates the models, applies them to the rest of the rows and saves them to disk. This yields a separate anomaly score for each field (csv column) at each time step, which can then be combined, thresholded or otherwise analyzed.

I made it for my own research involving n-dimensional movements but generalized it with a config file, so you give it the file paths, N first rows to sample from, whether you want 1 multivariate model or multiple univariate and a couple other things.

Right now I don’t have it shared on Github, though if you’d like to test it out and would cite the repo I’d consider sharing it.

Also @pizzatakeaway (funny handle), I glanced at your work and noticed this:

> the algorithm was not able to tell which feature was responsible

I think the multiple univariate model approach should help with this, since you get separate anomaly scores for each model at each time step.

Also have you seen NAB (Numenta Anomaly Benchmark)? In case not I’d definitely recommend checking it out. Its purpose is to test HTM against other anomaly detectors including Etsy Skyline and Twitter ADVec across a range of data sets from numerous domains. I don’t think it includes RNN or LSTM as you do though. The code actually has a ‘create\_new\_detector’ script too, so you could potentially plug in your RNN or LSTM detector to it. That’d allow you to quickly test those or any detectors over like 55+ data sets, which already have labeled anomalies and built-in detector-scoring functions.

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 23, 2020, 10:08am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/25 "2020-04-23T10:08:55Z")

</div>

@steinroe sure, here is the whole document: [https://buttignon.next-cloud.org/index.php/s/a52D5bxBSMKqSNn](https://buttignon.next-cloud.org/index.php/s/a52D5bxBSMKqSNn)

@sheiser1 Nupic or _multiple univariate model_ if you prefer call it that way, **it has been outperformed by a multivariate HTM system** in the experiment I run. Why? Because the algorithm benefits from the correlations in the data. The ECG Experiment (the first one) shows it clearly. Nupic recognizes the anomaly on the 1st signal, but not on the second.

Furthermore, _reading at glance_ misled you. **The Feature Importance method worked very well**.  
The meaning of the whole sentence was:

> So far, performing multivariate anomaly detection with HTM was possible but incomplete.  
> The algorithm was not able to tell which feature was responsible for the anomaly, and to  
> which extent.

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 23, 2020, 10:33am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/26 "2020-04-23T10:33:33Z")

</div>

> [@sheiser1](#):
>
> Also have you seen NAB (Numenta Anomaly Benchmark)? In case not I’d definitely recommend checking it out. Its purpose is to test HTM against other anomaly detectors including Etsy Skyline and Twitter ADVec across a range of data sets from numerous domains. I don’t think it includes RNN or LSTM as you do though. The code actually has a ‘create\_new\_detector’ script too, so you could potentially plug in your RNN or LSTM detector to it. That’d allow you to quickly test those or any detectors over like 55+ data sets, which already have labeled anomalies and built-in detector-scoring functions.

NAB is a great tool, but there are only univariate datasets to be found.  
In try to find multivariate datasets e.g. the Damadics dataset cited by Subutai at the end of one of its papers in the Future Works paragraph.  
On multivariate datasets I applied Nupic (I called it _uHTM_ that stands for _univariate HYM_), Etsy Skyline and Twitter ADVec on each signal singularly: I loaded each time series on NAB and let the script do the threshold fine tuning.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 24, 2020, 12:50am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/27 "2020-04-24T00:50:34Z")

</div>

> [@pizzatakeaway](#):
>
> **outperformed by a multivariate HTM system** in the experiment I run. Why? Because the algorithm benefits from the correlations in the data.

Makes sense yes, I’ve experienced this also. I think the single multivariate model approach is better for this reason, as long as there aren’t too many variables to squeeze into it.

> [@pizzatakeaway](#):
>
> **The Feature Importance method worked very well**.

I’m glad to hear it! It makes me curious, if the algorithm “was not able to tell which feature was responsible for the anomaly”, how were the features measured for importance?

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 6:19am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/28 "2020-04-24T06:19:15Z")

</div>

> [@sheiser1](#):
>
> Makes sense yes, I’ve experienced this also. I think the single multivariate model approach is better for this reason, as long as there aren’t too many variables to squeeze into it.

Yeah most probably. It should be researched further. I stopped to 6-7 variables (6 was semi-supervised, 7 unsupervised).

For example, in this experiment (ECG Fibrillation), 2x univariate HTM has been applyed the two channels.  
According to them, only in the first channel the anomaly was correctly detected, and not in the second.

 ![grafik](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/1/1a43ccdb9254a3805396817c71e0756494763b7a.png)

> [@sheiser1](#):
>
> I’m glad to hear it! It makes me curious, if the algorithm “was not able to tell which feature was responsible for the anomaly”, how were the features measured for importance?

Actually was pretty simple (once I realized it), I expained it in 4.4 and you can see it in action in Chap 5.

_To work with multivariate time series, we need to encode every single feature and stack the single encodings upon each other to form the binary input vector. The key is to keep track of the indices —x\_i to x\_j— of the single encoding vectors._  
_When the input vector successively enters the SP, even though the active bits change as shown in the Fig, the c\_ij cell in SP is always representing the x\_i cell in the input SDR._  
_Therefore, we can travel back from TM to x\_i , and identify to which encoded feature x\_i belongs to. For the same reason, moving from the SP embedding to the TM embedding is possible without losing the traceability of our input x\_i._  
_In the next step, the AS retrieves from TM not only the information about how many active columns were not predicted, but also which ones. When the anomaly likelihood for an instance, exceeds the threshold θ\_lh, we use the indices of the active not predicted columns to access the input vectors. Finally, by counting the overlapping indices with each input vector, we calculate which features caused the anomaly and to which extent._

 ![grafik](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/0/0a8b481fee831ac4e438e15bc6fe329d3abef365.png)

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [April 24, 2020, 6:52am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/29 "2020-04-24T06:52:30Z")

</div>

@pizzatakeaway for ECG AD there is another method for classifying QRST pattern so that you can classify SDR as anomaly. I am very interested in knowing how better your method is than using SDR-classification.  
Do you have any comparison result? Thanks

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 8:41am UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/30 "2020-04-24T08:41:54Z")

</div>

Hi @thanh-binh.to, here is all I got: [https://buttignon.next-cloud.org/index.php/s/a52D5bxBSMKqSNn](https://buttignon.next-cloud.org/index.php/s/a52D5bxBSMKqSNn)  
There is code and more graphs, but the repo is not very user friendly at the moment 😕

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [April 24, 2020, 12:04pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/31 "2020-04-24T12:04:16Z")

</div>

@pizzatakeaway what do you mean „code“?  
Your algorithm description? Or SW code?

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [April 24, 2020, 2:12pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/32 "2020-04-24T14:12:06Z")

</div>

@pizzatakeaway: I am very interested in how you do your experiments with mHTM and LSTM for each data in datasets:

1. Do you train LSTM before? or
2. do you run both HTM and untrained LSTM and consider AS they output?
3. or train them from T=0 till N, I say N = 1000 ECG-points and after that turn them to inference mode for AD for all remain ECG points?

Thanks

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 3:58pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/33 "2020-04-24T15:58:24Z")

</div>

@thanh-binh.to hi again 🙂  
LSTM has been implemented according to the architecture presented in the paper of [Malhotra et al. 2015](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-56).

1. Yes. In the graphs there is a gray shaded area. This area represents the evaluation set and it divides the training from the test set. In the training and aevaluation set there are no anomalies.
2. for HTM it means I use the training set to feed the LH. Of course I wait for the AS to settle down.
3. The learning modus by HTM is always on.

Results and evaluations are only take the TEST set into account.  
Did I answer your questions properly?

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 4:00pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/34 "2020-04-24T16:00:25Z")

</div>

I mean the Python code I wrote to implement the feature importance method and run all the experiments presented in the paper.

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 24, 2020, 4:25pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/35 "2020-04-24T16:25:20Z")

</div>

> [@pizzatakeaway](#):
>
> the AS retrieves from TM not only the information about how many active columns were not predicted, but also which ones. When the anomaly likelihood for an instance, exceeds the threshold θ\_lh, we use the indices of the active not predicted columns to access the input vectors. Finally, by counting the overlapping indices with each input vector, we calculate which features caused the anomaly and to which extent.

Ah I see, that’s great!  
So you keep track of which encoded features all the SP columns represent, then you can see how much each feature contributes to each anomaly score above 0.  
**Do you have this implemented using NuPIC models?**

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [April 24, 2020, 6:08pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/36 "2020-04-24T18:08:02Z")

</div>

@pizzatakeaway thanks for your answer! I am not sure if the results are comparable, because HTM is always On so that it learn too the abnormal QRS pattern!  
Please correct me if I am wrong. I think, each data has 2 parts: training and inference. Both HTM and LSTM learn only with train part and than turn to inference mode for anomaly detections!

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 6:37pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/37 "2020-04-24T18:37:27Z")

</div>

> [@sheiser1](#):
>
> Do you have this implemented using NuPIC models?

What do you mean with NuPIC models?

---

<div class="post-metadata">

**Author:** ![sheiser1](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/sheiser1/32/766_2.png) [@sheiser1](https://discourse.numenta.org/u/sheiser1)\
**Post date:** [April 24, 2020, 7:08pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/39 "2020-04-24T19:08:02Z")

</div>

I mean how did you generate your HTM models? Was it with NuPIC or some other framework? Or you own home-built version? I’d love to see how you implemented the feature importance system you eloquently presented there!

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 7:10pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/40 "2020-04-24T19:10:06Z")

</div>

Sure… it is one of the “downsizes” of HTM. They can learn new patterns but if anomalous patterns are recurrent, they end up with learning them.  
On the other side we can philosophize on the meaning of “anomalous/anomaly”: if an anomaly is recurrent, is not an anomaly anymore but rather an unwanted behavior. In this case HTM as it is now, is not the go-to algorithm.

Concerning your question:

1. TRAIN + EVALUATION SET (normal data)

- HTM (`learn`:on): AS decreases, at some point we start feeding the LH and the Threshold is set
- LSTM: try to learn by minimizing the loss function. The prediction errors are used to feed a multivariate Gaussian. Here the LH corresponds to the logarithm of the Probability Density Function (PDF) derived from the multivariate Gaussian distribution. The Threshold is set

1. TEST SET (normal data + anomalies)

- HTM (`learn`:on): LH evolves with the AS
- LSTM: LH is dynamic (feeded by the Prediction errors)

---

<div class="post-metadata">

**Author:** ![pizzatakeaway](https://avatars.discourse-cdn.com/v4/letter/p/9d8465/32.png) [@pizzatakeaway](https://discourse.numenta.org/u/pizzatakeaway)\
**Post date:** [April 24, 2020, 7:17pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/41 "2020-04-24T19:17:04Z")

</div>

Ok, I got it now.

I used the Nupic functions yes! Scalar Encoders, SP, TM and LH.

I just played around with the, LH, AS and the information that are also available in the verbose results: which cells were predicted in (t-1) which are on now in (t) and which are the **unpredicted cells in (t)**.  
Combining this last info with the indexes of the input vectors did the magic.

---

<div class="post-metadata">

**Author:** ![JWaldhorn](https://avatars.discourse-cdn.com/v4/letter/j/7bcc69/32.png) [@JWaldhorn](https://discourse.numenta.org/u/JWaldhorn)\
**Post date:** [December 7, 2020, 4:21pm UTC](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316/42 "2020-12-07T16:21:47Z")

</div>

Hi @pizzatakeaway I came across this thread while searching for multivarite time series in htm and wanted to read your work. Unfortunately the links only lead to error messages. Is the work published in the meantime (could not find anything) or could you send me a link? I would appreciate it very much.

[Previous page](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316.md?page=1)

[Next page](https://discourse.numenta.org/t/can-htm-be-used-on-multivarient-time-series-problems/3316.md?page=3)