NPE during org.numenta.nupic.network.Layer deserialization

Engineering HTM.Java
1

Hello,

There is one more problem with the HTM deserializaton.
We use the SerializerCore to serialize/deserialize HTM classes.
SerializerCore htmSerializer = new SerializerCore(SerialConfig.DEFAULT_REGISTERED_TYPES);

During deserialization the NPE occurs on the code publisher.onNext(t) in the method Layer.compute(T t).

It seems that the publisher field is null.
It is declared as transient in the Layer calss and need to be initialized in the method postDeSerialize().
however the postDeSerialize() method has the logic:

    if(sensor != null) {
        sensor.setLocalParameters(params);
        // Initialize encoders and recreate encoding index mapping.
        sensor.postDeSerialize();
    }else{
        // Dispatch functions (Observables) are transient & non-serializable so they must be rebuilt.
        observableDispatch = createDispatchMap();
        // Dispatch chain will not propagate unless it has subscribers.
        parentNetwork.addDummySubscriber();
    }

It seems that if sensor is null then the publisher field is initialized during the createDispatchMap() call.
But if the sensor is not null the publisher is not initialized.

The used version is org.numenta html.java 0.6.13.

1 Like
2

Hi @alexsch,

Not to put off your question but, let’s resolve the other issue and then return to this one once we make sure you’re using the correct version of the library?

In the meantime, something to think about may be…

The correct way to use the Publisher is to use the newer PublisherSupplier class which was created for use with the PersistenceAPI - (as excerpted from class documentation)

The old way of creating Publishers has now changed when the {@link PAPI} is used.
Instead, a {@link PublisherSupplier} is used, which ascribes to the {@link Supplier} 
interface which acts like a "lazy factory" and kicks off other needed settings when a new
Publisher is created...

…that may be something to look at (I’m not sure if you’re using that class)?

============
Re: SerializerCore usage…

You have to be careful with this one because using the SerializerCore bypasses other methods which “support” the persistence of HTMs within the Network API framework. Please refer to the tests to make sure your serialization is correct…

Here is one example of Network serialization (though there are many). Have a look at the PersistenceAPITest and the NetworkTest class.

Hope this helps…

3

We use the org.numenta:htm.java:jar:0.6.13 which has a reference to the algorithmfoundry:algorithmfoundry-shade-culled:jar:1.3:compile

We used the SerializerCore just because we thought that it calls methods from the Persistable interface.

Is it possible to rely on the SerializerCore if it is obtained from the PersistenceAPI like it is descibed in the
Saving-Your-Network:-The-Persistence-API

Our use case is the following. We use the Kryo serializer in the Spark Streaming application and we need to store the serialized HTM objects in the Kryo.

Probably we can use the following code from the suggested PersistenceAPITest to save and restore the HTM classes in the Kryo Input/Output:

        Parameters p = getParameters();
        
        SerialConfig config = new SerialConfig("testSerializeParameters", SerialConfig.SERIAL_TEST_DIR);
        
        PersistenceAPI api = Persistence.get(config);
        
        // 1. serialize
        byte[] data = api.write(p, "testSerializeParameters");
        // save data into the Kryo Output


        // 2. deserialize
        // read data from the Kryo Input
        Parameters serialized = api.read(data);
4

I have one more question: about the SerializerCore.serialize(T instance)/deSerialize(byte[] bytes) methods.

The deSerialize(byte[] bytes) method calls the Persistable.postDeSerialize() method:

But the serialize(T instance) does not call Persistable.preSerialize():

Is it consistent behavior?