Question about serialization behaviour

[Finn]

Hello,
I am currently working on a BA thesis with the topic “Hierarchical temporal memory for in-car network anomaly detection”. I am nearing the end of my work and I have noticed a few things so far, some of which I can’t answer by simply investigating the issue. So I hoped that someone here can help me out.

I am using HTM-core 2.1.15 with python 3.7 and based my program off the hotgym example. I am analyzing 4 metrics (jitter, avg. gap between packets, bandwidth and frequency for 10ms timeframes) encoded into the same SDR and it works pretty well on our TSSDN network.

One of my main confusions though is that the de-/serialization functions for loading and storing TM/SP (loadFromFile/saveToFile) produce very different results depending on if the model is trained, then stored and loaded again (with learning turned off afterwards) or simply running in online unsupervised learning mode the whole time.
After having done many tests it seems like the TM is the main culprit as it needs a few hundred iterations so that it doesn’t just oscillate in anomaly score the whole time. Another observation is that the anomaly score is waaay less sensitive to the data after loading. Let me show you a few pictures to explain.

First an image of anomaly score oscillating when TM is reloaded but not learning for enough iterations before learn=False (forgive me for the many different values shadowing each other but you should get the gist):

now for the live-reloaded comparison:
This is normal learning with 3 anomalies (DoS attacks) in the data.

This is the reloaded TM/SP fed with the same data and TM gets 900 startup iterations where learn=True

I have done other tests where I learned on a clean dataset with no anomalies and reloaded + analyzed the dataset including anomalies afterwards. It ran on different parameters, was showing VERY much noise on the learning set while showing adequate results after reloading. Same here is that it somehow gets desensitized after loading:

Learning on clean dataset (green values are anomaly raw score, used it for testing/comparing to tm.anomaly)

Analyzing anomalous data after reloading

I am not sure if the parameters would be any help as it is always the same difference in sensitivity it seems and my main question is: Why is there any difference at all, is the serialization working properly or is there something going missing in the process?

Ofc for completion purposes I will provide you the lines that load/store the TM/SP:

storing:
self.sp.saveToFile(_TEST_DIR + '/sp_' + timestring + '.tmp') self.tm.saveToFile(_TEST_DIR + '/tm_' + timestring + '.tmp')

loading:

sp_tmp = SpatialPooler()
SpatialPooler.loadFromFile(sp_tmp, _TEST_DIR + '/sp_' + self.parameters["application"]["model"] + '.tmp')

tm_tmp = TemporalMemory()
TemporalMemory.loadFromFile(tm_tmp, _TEST_DIR + '/tm_' + self.parameters["application"]["model"] + '.tmp')`

[dmac]

@Finn

Why is there any difference at all, is the serialization working properly?

Hi Finn,

A few things:

1. This sounds like a bug, serialization is probably not working correctly. All things equal, saving and then loading the model should not change the results. To be clear, you’ve tested it with NO save/loading and it works correctly?

2. Have you tried a installing the HTM.Core library from the source code?

Thanks for reporting this issue!

[Finn]

hey dmac, thanks for the reply.

to answer your questions:

This sounds like a bug, serialization is probably not working correctly. All things equal, saving and then loading the model should not change the results . To be clear, you’ve tested it with NO save/loading and it works correctly?

Yes, I have tested it without any storing/loading. The results of such a test can be found in the 2nd picture of my 1st post. I am willing to use online learning only without storing, but it’s a matter I would have liked to discuss in my work since both options have their advantages/disadvantages.

Have you tried a installing the HTM.Core library from the source code?

No I haven’t, I installed the library from the 2.1.15 .whl from the release page on github and have just checked whether there were any substantial changes in the TM files since then. It seemed like there weren’t any changes that would impact this behaviour but I could have overlooked something.

[dmac]

I can not reproduce this bug.

Starting the latest source code from the github page, I modified the hotgym example:

1. I added the seed parameter to all algorithms so that it should be 100% deterministic.
2. I added the following snippet of code to save & load the TM to/from file halfway through the test:

    if count == 1000:
      tm.saveToFile("tm_save.temp")
      tm2 = TemporalMemory()
      tm2.loadFromFile("tm_save.temp")
      tm = tm2

And this yields identical results, both with and without save/loading the TM. The results are the same, even all of the floating point numbers which are printed at the end of the program:

Predictive Error (RMS) 1 steps ahead: 8.141706441154033
Predictive Error (RMS) 5 steps ahead: 8.945421905050706
Anomaly Mean 0.03885156511594449
Anomaly Std 0.1481963840587836

[dmac]

@Finn

substantial changes in the TM files

BTW, the TM algorithm is split between the TemporalMemory class and the Connections class.

Again, I would recommend trying to install the HTM.Core library from the source code. If that fixes your issue then tell us, and we will do another release with the latest sources.

[Finn]

Forgot to mention that but I restart the application between runs in my example.
I have tried reloading it during a single run too once. When I did that it didn’t make any difference to the result either, as if it was never reloaded. Which is weird because you’d think that Python properly gets rid of any stored TM data by reassigning it the way you did.

I have found a recent issue on github where another user states a similar problem but it’s hard to understand the way he tested it:

Question about a method in the spatial pooler and temporal memory in algorithm

[dmac]

@Finn

I restart the application between runs in my example.

Yes, now I am able to reproduce the bug!

EDIT: Actually, this bug is a bit more complicated…

[dmac]

To reproduce this bug, I followed this procedure:

1. Create a full HTM system (encoder, spatial pooler, temporal memory, classifier, anomaly tracker).
2. Run the first half of the hotgym dataset through the HTM.
3. Save the HTM to file. ← Problem is here!
4. Exit the currently running process and restart python.
5. Load the HTM from file.
6. Run the second half of the hotgym dataset through the HTM. Observe that the HTM does not perform as well during the second half of the dataset as it did for the first half.

The problem here is that the HTM can not be 100% saved to file. The Predictor and AnomalyLikelihood can not be saved to file despite the fact that they contain learned data. My guess is that you’re using a fully trained HTM with a brand new (untrained) classifier and AnomalyLikelyhood class.

Those classes are capable of being serialized in C++, but there are no python bindings for save/loading those classes. This is an oversight, and it should be relatively straightforward to fix.

[thanh-binh.to]

@dmac I remember that we had the serialization problems 2 years ago but we did not solve it.
I found it by testing MNIST applications. If I run inference directly after training, I get higher scores than I save model after learning, and then load it for inference

[dmac]

I’d forgotten that most of the htm.core library implements the pickle protocol, so you can serialize all of those classes. pickle — Python object serialization — Python 3.7.10 documentation I was able to do this and show that the bug is fixed (at least for me).

import pickle

# Save the entire HTM system to file.
with open("save.tmp", "wb") as f:
    htm_system = (sp, tm, predictor, anomaly_history)
    pickle.dump(htm_system, f)

# Load the entire HTM system from file.
htm_system = pickle.load(open("save.tmp", "rb"))
(sp, tm, predictor, anomaly_history) = htm_system

I hope this fixes your problem!

[Finn]

wow I didn’t expect such a simple solution

I have already decided on using online learning mode going forward but I will add it for completion purposes, thank you very much.