Testing NAB with HTM.java with different configuration

Engineering NAB
,
1

Hi All,

I am interested in performing some experiments with different parametric configuration of HTM java on NAB to find out what would be the possible impact on the NAB scores and what parameters of HTM java can give the best scores. For establishing experimental setup to get suitable parameters i want to use genetic algorithm and treating NAB as a fitness function. I have studied NAB and its scoring mechanism , Following are few questions that are in my mind regarding NAB and HTM java

1- I have seen NAB have 3 score (standard, Reward low FP, Reward low FN). According to my understanding standard scores are those which NAB gives algorithm on identifying TP and TN during anomaly window. FP score should be related to false alarms. But NAB gives this score on producing false alarm or not producing false alarms? Same question for FN it should be related to no alarm on anomaly But NAB gives this score to algorithm on finding anomaly or not finding? (these score are basically a credit or dis-credit to be high?)

2- HTM have lots of configurations and I assume all configurations have their own significance as mentioned in HTM manuals. But some configurations would be most critical and they have a strong impact on results. According to my understanding three parameters related to architecture (Input dimension, column dimension, cell per column) are most important. Please suggest me what could be other critical parameters should i consider in my experiments?
(I have performed few experiments on above mentioned three parameters and i got different results)
Please also guide my any other point regarding my understanding and experiment.

@cogmission can you please help me regarding above mentioned queries?
Also I have noticed that when i run on default configurations htm java gives me below scores.

image
image.png1091Ă—477 302 KB

But i have seen below scores in white paper of NAB.

image

Is this a normal difference of htm java or there is some particular issue in my system.

I am also attaching htm java default parameters. I have print it in a file.

image
image.png570Ă—607 11.1 KB

Regards,
@Usama_Furqan

2

Hi @Usama_Furqan,

I’m sorry, I can’t really help you analyze your differences as these algorithms are created by Numenta - I just re-created the underlying algorithms in Java. The NAB itself (since it was written by Numenta), and the system behavior are their “domain”. My primary purpose was to ensure that identical behavior is elicited for the underlying algorithms and for commensurate use of the allegorical Network API. For help with the Java version of “NAB”, I would maybe consult @lscheinkman since he created it; and Numenta engineers since they created NuPIC. Otherwise, I’m learning about the algorithm behavior like everyone else here, and don’t claim to be an expert! :slight_smile:

The Numenta NAB has a few “tweaks” one of which may be (though I’m not quite sure), use of the “old back tracking TemporalPooler” algorithm. But in any case, there are some tweaks that I don’t think have been recreated in the Java version of NAB (though the underlying Algorithms are perfectly equal). I would maybe ask your questions in the NuPIC forum as the language (minus consideration of the tweaks) is not important.

I’m sorry that I am unable to provide more insight as to the inner workings, but like I said, I’m learning this stuff like everyone else.

3

Hi @cogmission

I have another question related to HTM.java and NAB. As i know NAB contains 58 data sets. When we run NAB with htm.java , it creates one single HTM for all data sets or it creates separate HTM for every domain and data set?

According to theory there should be one HTM and we could run it for all data sets and for all domains. Like a human brain we don’t need to change any thing before learning different domain.

Regards,
@cogmission

4

The models all start with the same set of parameters, not tuned for each data set. But there is a model for each data set. Each model learns one data set. If you had one model learn them all, it would get confused without some context to distinguish them.

2 Likes