Questions about using saved models

[juanhorta]

Hi everyone,

Right now we’re trying to use previous trained models for anomaly detection. So after the training with a revised dataset, we disabled learning and saved the model.

model = createAnomalyDetectionModel(dataFrame)
     [...]
model.disableLearning()
model.save('/scalarModels/model_params/RequestCount.model')

After that we load the model and we test it with a real dataset but we’ve noticed that first values (around 400) aren’t evaluated, instead always have the same anomaly likelihood (0.5). It looks like we are not using the saved model and we need to learn again.

trained_model = ModelFactory.loadFromCheckpoint('/scalarModels/model_params/RequestCount.model')
trained_model.disableLearning()

Do Nupic need a time to wake up? Maybe I’m wrong but I thought that anomaly detection based in a trained model would work since the first value.

Regards,

Juan

[rhyolight]

You’ve found a deficiency in NuPIC. Currently, saving a model in the OPF will not save the state of the anomaly likelihood because it is a post-process. I could probably pickle the AnomalyLikelihood instance yourself and manually resurrect it with the model and re-attach.

[vkruglikov]

The anomaly service in htmengine (from numenta-apps repo) deals with this by saving the anomaly state in mysql as part of its normal processing of many models.

[juanhorta]

Hi,

thanks a lot for your help, I will read the HTM Engine code.

I’m trying to build the next schema:

json input --> Trained Nupic Docker --> ElasticSearch <-- Kibana

A more simplified schema than the HTM Engine, without RabbitMQ, neither mysql.

At the first step, I was using HTM Engine but I was not able to use a complex model.

https://discourse.numenta.org/t/advanced-settings-for-model-parameters-in-htm-engine/1366/5?u=juanhorta

Regards.

[vikash0837]

Hi @juanhorta thanks for you help. I have implemented anomaly likehood almost as the method suggested by @rhyolight . I am pickling one month data which include timeStamp ,raw anomaly score and scalar input value. I have modified anomaly_likelihood.py and feeding the pickled value when calling anomalyProbability() function. I am updating the pickled value each time i am coming out of anomalyProbability() function. I am able to get the result without having the issue of having cold start for 1st 388 value. Still i need to verify the results.If you need some more information reach me out @vikash0837@gmail.com.

[Pegasus]

Hi, Taylor, I have been the save/load stage, now. And there are some questions:

@juanhorta

Do Nupic need a time to wake up? Maybe I’m wrong but I thought that anomaly detection based in a trained model would work since the first value.

1. Does the ‘deficiency’ mentioned in this topic has be repaired ?, if not, you mean use the module ‘pickle’, right? As I am not familiar with this module, could you give me some more information about how to save the likelihood by using pickle ?

2. I have got the right place to learn how to save/ load models: http://nupic.docs.numenta.org/1.0.0/guides/serialization.html
   However, I am a little confused with it. As I have read the code of htm_prediction ( and model_factory), I have find there are two ways to save/load model:

• save/load: inside the code, it take advantage of pickle module.

• readFromCheckpoint/writeToCheckpoint ( or loadFromCheckpoint of model_factory, which inner call readFromCheckPoint ): behind them is capnp

  Both of these have been deprecated, according to the page: “Serialization” (i.e., the url mentioned above). Since the methods are still in the the model, does it means I could still use them either ? then, I wonder the difference between these two ways with respect to the saved model ?

3. In the page mentioned above(i.e., the url above), writeToFile/readFromFile are the recommended methods, but in the htm_prediction model, I could not find the functions, does it means I can’t still use them? If they could be used now, does it work similarly to the methods mentioned above. Say I would like to save a model I can just work like this: model.writeToFile(dir) ? as the example just illustrates how to save SP, I don’t if it is the same to model.

4. I find a topic: “Load/Save model may lose precision?”
   Load/Save model may lose precision?
   Does this affect much to the result or model?

As last, I there is another question
5 . HTM loss its prediction ability - #13 by Pegasus
In this topic, I ask a question, could you give an answer ?

@Pegasus

The HTM as often the case fails to predict when anomaly occur, because the likelihoods of its predictions are ridiculous then abandoned, instead of the real real data at that time.

Do I get the point? I refer to the source code, there indeed are so called ‘best prediction’ but I couldn’t find the logic of mimicking raw data, and I wonder how the likelihoods of prediction( not the likelihood value of anomaly) are calculated ? Does the process of calculation take advantage of the real data in advance? Still, I have no idea where to get these information in the source code.

Thanks

[rhyolight]

@Pegasus

Does the ‘deficiency’ mentioned in this topic has be repaired ?, if not, you mean use the module ‘pickle’, right? As I am not familiar with this module, could you give me some more information about how to save the likelihood by using pickle ?

No, it is not repaired, and we don’t plan on repairing it. You can pickle the Anomaly components of your models yourself. Just look up how to pickle objects with python.

@Pegasus

Since the methods are still in the the model, does it means I could still use them either ? then, I wonder the difference between these two ways with respect to the saved model ?

Yes, you can still use them. The new way to serialize is more granular, meaning you save smaller objects like algorithm instances. The old way using pickle will pickle the entire model object, including the algorithm instances. If you use the old way it should still work.

@Pegasus

In the page mentioned above(i.e., the url above), writeToFile/readFromFile are the recommended methods, but in the htm_prediction model, I could not find the functions, does it means I can’t still use them?

Right, that model is not serializable using the new way. Only the algorithm instances themselves are serializable. If you are using HTMPredictionModel in the OPF, you might find it much easier to use the old way, but it will be slower.

@Pegasus

Load/Save model may lose precision?

Does this affect much to the result or model?

I don’t totally understand that issue. You might want to comment there. It does seem like there is a bug there, and I filed a ticket about it, but not sure it will get worked on soon.

[Pegasus]

Thanks, Taylor, got it. you have, again, resolved my puzzles. And I have saved several models now, I will try to save likelihood by using pickle. Thanks a lot, again, sincerely.

[Purestare3]

Hi, sorry for taking the liberty of bothering you. I am wondering can I inquire you about how to save/reload trained model of HTM nupic library? I am recently working on a project that needs to store the model state and came across some issue when saving the trained model. But when I saw your post on HTM forum saying that you successfully saved and reload the model, I felt very excited and curious about how you save the model or what is the environment or procedure that you save the model. I’d appreciate any instruction/suggestion/reference from you and thank you so much for your time and patience!

FYI, I did the following lines of code to save the trained OPF model:

model.disableLearning()
model.save('/Users/ruzhong/Documents/python/HTM_editor/OPF_cert_category.model')

And then error shows up like following:

Traceback (most recent call last):

  File "test.py", line 413, in <module>

    model.save('/Users/ruzhong/Documents/python/HTM_editor/OPF_cert_category.model')

  File "/Users/ruzhong/Library/Python/2.7/lib/python/site-packages/nupic/frameworks/opf/model.py", line 360, in save

    self._serializeExtraData(extraDataDir=self._getModelExtraDataDir(saveModelDir))

  File "/Users/ruzhong/Library/Python/2.7/lib/python/site-packages/nupic/frameworks/opf/htm_prediction_model.py", line 1429, in _serializeExtraData

    self._netInfo.net.save(outputDir)

  File "/Users/ruzhong/Library/Python/2.7/lib/python/site-packages/nupic/engine/__init__.py", line 729, in save

    engine_internal.Network.save(self, *args, **kwargs)

  File "/Library/Python/2.7/site-packages/nupic/bindings/engine_internal.py", line 1214, in save

    return _engine_internal.Network_save(self, *args, **kwargs)

SystemError: NULL result without error in PyObject_Call

Thanks again for your time and patience!! I’d appreciate it if you can reply to me at your convenience.

[Purestare3]

Hi may I know how did you saved your model? I am currently having some issues wrt saving OPF model as I indicated in my last reply to @vikash0837 in this post. Thanks for your time and patience and I will appreciate your help!

[Purestare3]

I’d also appreciate any suggestions from @juanhorta on saving/reload the model state since I am having some issues wrt saving model state recently (plz refer to my reply with error trace to @vikash0837 ). Thanks!

[rhyolight]

@Purestare3 Just wanted to make sure you saw the serialization guide in the API docs.