Spatial Pooler Serialization problem - too big


I have stored a Spatial Pooler as follows:

with open(“out_sp.tmp”, “wb”) as f1:

Which is fine, but I cannot re-load it using:

with open(“out_sp.tmp”, “rb”) as f1:
sp = SpatialPooler.readFromFile(f1)

I get the following error message:

  sp = SpatialPooler.readFromFile(sp1)
  File "/usr/local/lib/python2.7/dist-packages/nupic-1.0.6.dev0-py2.7.egg/nupic/", line 94, in readFromFile
    proto = schema.read_packed(f)
  File "capnp/lib/capnp.pyx", line 2962, in capnp.lib.capnp._StructModule.read_packed (capnp/lib/capnp.cpp:61515)
  File "capnp/lib/capnp.pyx", line 3554, in capnp.lib.capnp._PackedFdMessageReader.__init__ (capnp/lib/capnp.cpp:69069)
capnp.lib.capnp.KjException: capnp/serialize.c++:197: failed: expected totalWords <= options.traversalLimitInWords; Message is too large.  To increase the limit on the receiving end, see capnp::ReaderOptions.

The above error message suggests setting the “ReaderOption” . I have figured out what needs to be done (set the option called “traversalLimitInWords” (which has a default value of 8 * 1024 * 1024) – but I cannot figure out how to change it? (I have figured out that capnp is a C++ program but I cannot work out how to set this parameter from python?)

What is the normal size of a Spatial Pooler when stored by this method? Mine is 83MB, do I need to make it smaller by reducing the size of my input SDRs? (The SP has 2048 columns.)



Unfortunately, I’m not sure that pycapnp even exposes ReaderOptions, see doc search.

Try fewer mini-columns. Instead of 2048, try 1024. It may perform just as well.