Using HTM with Kryo serializer

Engineering HTM.Java
1

Hello,

My question just arises from the topic NPE during org.numenta.nupic.network.Layer deserialization

I need to serialize HTM classes using Kryo serializer which used in Spark Streaming application.
The serialization code looks like:

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.Serializer;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import org.numenta.nupic.model.Persistable;
import org.numenta.nupic.network.Network;
import org.numenta.nupic.serialize.SerialConfig;
import org.numenta.nupic.serialize.SerializerCore;

public class SparkKryoHTMSerializer<T extends Persistable> extends Serializer<T> {
    private final SerializerCore htmSerializer = new SerializerCore(SerialConfig.DEFAULT_REGISTERED_TYPES);

    @Override
    public void write(Kryo kryo, Output kryoOutput, T t) {
        T preSerialized = t.preSerialize();
        byte[] serialized = htmSerializer.serialize(preSerialized);
        kryoOutput.writeInt(serialized.length);
        kryoOutput.writeBytes(serialized);
    }

    @Override
    public T read(Kryo kryo, Input kryoInput, Class<T> aClass) {
        int size = kryoInput.readInt();
        byte[] bytes = kryoInput.readBytes(size);
        return htmSerializer.deSerialize(bytes);
    }
}

Should the PersistenceAPI be used instead of just the SerializerCore?
If so, is it possible to avoid to save HTM on disk and just get for each class its serialization byte array?

2 Likes
2

Hi @alexsch,

Thank you for using HTM.java!

I will take a look at this and get back to you shortly with some advice or a way forward…

Thanks!
David