Use htm for big data


#1

Hi,
I have a 1000 * 4000 matrix (it means 1000 vector of size 4000) and I like to give it to HTM to anomaly detection. can HTM predict on such big matrix? is there any limitation on the size of input?


#2

HTM, which the T stands for Temporal, works with temporally coherent data. If your matrix does not contain any element of time. HTM won’t work.


#3

Maybe he means the matrix itself changes state over time. Is this a weight matrix? :wink:

What data type does the matrix contain? Even if binary, an input space of 4,000,000 bits is too large for current HTM systems. You will have to find a way to reduce this data.


#4

yes, I have 1000 vector that are time series and each vector have 4000 element. I like to get 1000 vector to HTM and HTM detect anomaly .


#5

no I have 1000 vector and any of them changes over time (I have 1000 time series)
I like to detect anomaly of this 1000 vector by HTM. is this possible?
in your NYC Taxi passenger example there was 17000 vector (row) and each one have 4 element. I wanna to know if my element of time series increase (for example to 4000) can HTM predict yet?
What is the maximum size of input that HTM can work on it?


#6

If you can make the vector binary and it still retains meaning, I would say yes.

What example are you talking about? The only taxi data I know about is here and it does not look like that.


#7

If you have so many fields like that you could create separate models for each field, and then look for system anomalies as those times when many individual models are simultaneously showing anomalous behaviors.