Spatial Pooler Serialization problem - too big

Engineering NuPIC
1

I have stored a Spatial Pooler as follows:

with open(“out_sp.tmp”, “wb”) as f1:
sp.writeToFile(f1)

Which is fine, but I cannot re-load it using:

with open(“out_sp.tmp”, “rb”) as f1:
sp = SpatialPooler.readFromFile(f1)

I get the following error message:

  sp = SpatialPooler.readFromFile(sp1)
  File "/usr/local/lib/python2.7/dist-packages/nupic-1.0.6.dev0-py2.7.egg/nupic/serializable.py", line 94, in readFromFile
    proto = schema.read_packed(f)
  File "capnp/lib/capnp.pyx", line 2962, in capnp.lib.capnp._StructModule.read_packed (capnp/lib/capnp.cpp:61515)
  File "capnp/lib/capnp.pyx", line 3554, in capnp.lib.capnp._PackedFdMessageReader.__init__ (capnp/lib/capnp.cpp:69069)
capnp.lib.capnp.KjException: capnp/serialize.c++:197: failed: expected totalWords <= options.traversalLimitInWords; Message is too large.  To increase the limit on the receiving end, see capnp::ReaderOptions.

The above error message suggests setting the “ReaderOption” . I have figured out what needs to be done (set the option called “traversalLimitInWords” (which has a default value of 8 * 1024 * 1024) – but I cannot figure out how to change it? (I have figured out that capnp is a C++ program but I cannot work out how to set this parameter from python?)

What is the normal size of a Spatial Pooler when stored by this method? Mine is 83MB, do I need to make it smaller by reducing the size of my input SDRs? (The SP has 2048 columns.)

2

Unfortunately, I’m not sure that pycapnp even exposes ReaderOptions, see doc search.

Try fewer mini-columns. Instead of 2048, try 1024. It may perform just as well.