Serialization: writing and loading from disk a SpatialPooler object

Engineering NuPIC
,
1

Hi!

I’m quite new using the Python nupic package. I’m writing my first test programs and I need a way to save/load a SpatialPooler object to/from disk.

I have been googling for an answer, but I couldn’t find anything that works. I’m using the last release (1.0.4.dev0), and even thought there is documentation (http://nupic.docs.numenta.org/1.0.4.dev0/guides/serialization.html) stating that I should use the writeToFile/readFromFile methods, those methods simply don’t exist. And all the other options are deprecated. I have even tried with write/

I don’t know whether the problem comes from using the wrong version of nupic or what happens. I have read quite many answers to questions similar to mine, but none of them worked so far.

I would be very grateful if someone could help me with this. Thanks!

1 Like
2

Please paste the code you run to cause an error and the error you get when you try calling these functions on the sp instance.

1 Like
3

Thanks for your super quick answer.

Well as I do:

from nupic.bindings.algorithms import SpatialPooler

I get the error:

AttributeError: 'SpatialPooler' object has no attribute 'writeToFile'

because that class does not seem to have that method at all.

4

So you get that error not when attempting to call it, but when the class is loaded. @scott any ideas?

1 Like
5

@ivansiiito Can you confirm what version of nupic you are running? (Sometimes old artifacts stick around in python.)

1 Like
6

nupic 1.0.4.dev0
nupic.bindings 1.0.2

1 Like
7

Hi again!

I made a copy & paste mistake in my previous answer, and I don’t understand why I didn’t realize :-/

The code I wanted to paste was:

from nupic.algorithms.spatial_pooler import SpatialPooler
sp1 = SpatialPooler(inputDimensions=(10,), columnDimensions=(10,))
with open("out.tmp", "wb") as f:
  sp1.writeToFile(f)

And then I get:

AttributeError: 'SpatialPooler' object has no attribute 'writeToFile'

8

This script works for me with:

nupic 1.0.4.dev0
nupic.bindings 1.0.3.dev0

I think you need to install nupic.bindings 1.0.3.dev0 from source code.

1 Like
9

Hi again and thanks for your answer,

I’m a bit new with Python and I still have some problems installing external stuff. I have no problem to install nupic from sources (by the way, I’m using Windows). However, I cannot do the same with nupic.core (which I understand is what’s needed, since nupic.bindings is there).

I have been several hours reading and trying, uninstalling, downloading again, installing back… And I’m still unable to do it. I got the sources from https://github.com/numenta/nupic.core, and tried to do what’s said there. However, to start with, I cannot install pycapnp.

This is what I get when I try to install the dependencies:

C:\Users\ivans_000\Dropbox\3. Lenguaraz\nupic.core-master>pip install -r .\bindings\py\requirements.txt
Requirement already satisfied: numpy==1.12.1 in c:\python27\lib\site-packages (from -r .\bindings\py\requirements.txt (line 2))
Requirement already satisfied: pytest==3.0.7 in c:\python27\lib\site-packages (from -r .\bindings\py\requirements.txt (line 3))
Requirement already satisfied: pytest-cov==2.5.0 in c:\python27\lib\site-packages (from -r .\bindings\py\requirements.txt (line 4))
Requirement already satisfied: pytest-xdist==1.16.0 in c:\python27\lib\site-packages (from -r .\bindings\py\requirements.txt (line 5))
Collecting pycapnp==0.5.12 (from -r .\bindings\py\requirements.txt (line 6))
  Using cached pycapnp-0.5.12.tar.gz
Requirement already satisfied: colorama; sys_platform == "win32" in c:\python27\lib\site-packages (from pytest==3.0.7->-r .\bindings\py\requirements.txt (line 3))
Requirement already satisfied: setuptools in c:\python27\lib\site-packages (from pytest==3.0.7->-r .\bindings\py\requirements.txt (line 3))
Requirement already satisfied: py>=1.4.29 in c:\python27\lib\site-packages (from pytest==3.0.7->-r .\bindings\py\requirements.txt (line 3))
Requirement already satisfied: coverage>=3.7.1 in c:\python27\lib\site-packages (from pytest-cov==2.5.0->-r .\bindings\py\requirements.txt (line 4))
Requirement already satisfied: execnet>=1.1 in c:\python27\lib\site-packages (from pytest-xdist==1.16.0->-r .\bindings\py\requirements.txt (line 5))
Requirement already satisfied: apipkg>=1.4 in c:\python27\lib\site-packages (from execnet>=1.1->pytest-xdist==1.16.0->-r .\bindings\py\requirements.txt (line 5))
Building wheels for collected packages: pycapnp
  Running setup.py bdist_wheel for pycapnp ... error
  Complete output from command c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\ivans_~1\\appdata\\local\\temp\\pip-build-7beli4\\pycapnp\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d c:\users\ivans_~1\appdata\local\temp\tmpdbwcprpip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-2.7
  creating build\lib.win-amd64-2.7\capnp
  copying capnp\version.py -> build\lib.win-amd64-2.7\capnp
  copying capnp\_gen.py -> build\lib.win-amd64-2.7\capnp
  copying capnp\__init__.py -> build\lib.win-amd64-2.7\capnp
  copying capnp\__init__.pxd -> build\lib.win-amd64-2.7\capnp
  copying capnp\c++.capnp -> build\lib.win-amd64-2.7\capnp
  copying capnp\schema.capnp -> build\lib.win-amd64-2.7\capnp
  creating build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\helpers.pxd -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\non_circular.pxd -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\__init__.pxd -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\asyncHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\capabilityHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\checkCompiler.h -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\fixMaybe.h -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\rpcHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
  copying capnp\helpers\serialize.h -> build\lib.win-amd64-2.7\capnp\helpers
  creating build\lib.win-amd64-2.7\capnp\includes
  copying capnp\includes\capnp_cpp.pxd -> build\lib.win-amd64-2.7\capnp\includes
  copying capnp\includes\schema_cpp.pxd -> build\lib.win-amd64-2.7\capnp\includes
  copying capnp\includes\types.pxd -> build\lib.win-amd64-2.7\capnp\includes
  copying capnp\includes\__init__.pxd -> build\lib.win-amd64-2.7\capnp\includes
  creating build\lib.win-amd64-2.7\capnp\lib
  copying capnp\lib\capnp.pxd -> build\lib.win-amd64-2.7\capnp\lib
  copying capnp\lib\__init__.pxd -> build\lib.win-amd64-2.7\capnp\lib
  copying capnp\lib\pickle_helper.py -> build\lib.win-amd64-2.7\capnp\lib
  copying capnp\lib\__init__.py -> build\lib.win-amd64-2.7\capnp\lib
  copying capnp\lib\capnp.pyx -> build\lib.win-amd64-2.7\capnp\lib
  creating build\lib.win-amd64-2.7\capnp\templates
  copying capnp\templates\module.pyx -> build\lib.win-amd64-2.7\capnp\templates
  copying capnp\templates\setup.py.tmpl -> build\lib.win-amd64-2.7\capnp\templates
  running build_ext
  creating users
  creating users\ivans_~1
  creating users\ivans_~1\appdata
  creating users\ivans_~1\appdata\local
  creating users\ivans_~1\appdata\local\temp
  creating users\ivans_~1\appdata\local\temp\tmpojkh0v
  C:\Users\ivans_000\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG /Tpc:\users\ivans_~1\appdata\local\temp\tmpojkh0v\vers.cpp /Fousers\ivans_~1\appdata\local\temp\tmpojkh0v\vers.obj --std=c++11
  cl : Command line warning D9002 : ignoring unknown option '--std=c++11'
  vers.cpp
  c:\users\ivans_~1\appdata\local\temp\tmpojkh0v\vers.cpp(4) : fatal error C1083: Cannot open include file: 'capnp/common.h': No such file or directory
  *WARNING* no libcapnp detected or rebuild forced. Will download and build it from source now. If you have C++ Cap'n Proto installed, it may be out of date or is not being detected. Downloading and building libcapnp may take a while.
  fetching https://capnproto.org/capnproto-c++-0.5.3.1.tar.gz into c:\users\ivans_~1\appdata\local\temp\pip-build-7beli4\pycapnp\bundled
  error: [Error 2] El sistema no puede encontrar el archivo especificado

  ----------------------------------------
  Failed building wheel for pycapnp
  Running setup.py clean for pycapnp
Failed to build pycapnp
Installing collected packages: pycapnp
  Running setup.py install for pycapnp ... error
    Complete output from command c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\ivans_~1\\appdata\\local\\temp\\pip-build-7beli4\\pycapnp\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\ivans_~1\appdata\local\temp\pip-usnlti-record\install-record.txt --single-version-externally-managed --compile:
    Compiling capnp/lib\capnp.pyx because it changed.
    [1/1] Cythonizing capnp/lib\capnp.pyx
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-2.7
    creating build\lib.win-amd64-2.7\capnp
    copying capnp\version.py -> build\lib.win-amd64-2.7\capnp
    copying capnp\_gen.py -> build\lib.win-amd64-2.7\capnp
    copying capnp\__init__.py -> build\lib.win-amd64-2.7\capnp
    copying capnp\__init__.pxd -> build\lib.win-amd64-2.7\capnp
    copying capnp\c++.capnp -> build\lib.win-amd64-2.7\capnp
    copying capnp\schema.capnp -> build\lib.win-amd64-2.7\capnp
    creating build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\helpers.pxd -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\non_circular.pxd -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\__init__.pxd -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\asyncHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\capabilityHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\checkCompiler.h -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\fixMaybe.h -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\rpcHelper.h -> build\lib.win-amd64-2.7\capnp\helpers
    copying capnp\helpers\serialize.h -> build\lib.win-amd64-2.7\capnp\helpers
    creating build\lib.win-amd64-2.7\capnp\includes
    copying capnp\includes\capnp_cpp.pxd -> build\lib.win-amd64-2.7\capnp\includes
    copying capnp\includes\schema_cpp.pxd -> build\lib.win-amd64-2.7\capnp\includes
    copying capnp\includes\types.pxd -> build\lib.win-amd64-2.7\capnp\includes
    copying capnp\includes\__init__.pxd -> build\lib.win-amd64-2.7\capnp\includes
    creating build\lib.win-amd64-2.7\capnp\lib
    copying capnp\lib\capnp.pxd -> build\lib.win-amd64-2.7\capnp\lib
    copying capnp\lib\__init__.pxd -> build\lib.win-amd64-2.7\capnp\lib
    copying capnp\lib\pickle_helper.py -> build\lib.win-amd64-2.7\capnp\lib
    copying capnp\lib\__init__.py -> build\lib.win-amd64-2.7\capnp\lib
    copying capnp\lib\capnp.pyx -> build\lib.win-amd64-2.7\capnp\lib
    creating build\lib.win-amd64-2.7\capnp\templates
    copying capnp\templates\module.pyx -> build\lib.win-amd64-2.7\capnp\templates
    copying capnp\templates\setup.py.tmpl -> build\lib.win-amd64-2.7\capnp\templates
    running build_ext
    creating users\ivans_~1\appdata\local\temp\tmp9xtjg9
    C:\Users\ivans_000\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG /Tpc:\users\ivans_~1\appdata\local\temp\tmp9xtjg9\vers.cpp /Fousers\ivans_~1\appdata\local\temp\tmp9xtjg9\vers.obj --std=c++11
    cl : Command line warning D9002 : ignoring unknown option '--std=c++11'
    vers.cpp
    c:\users\ivans_~1\appdata\local\temp\tmp9xtjg9\vers.cpp(4) : fatal error C1083: Cannot open include file: 'capnp/common.h': No such file or directory
    *WARNING* no libcapnp detected or rebuild forced. Will download and build it from source now. If you have C++ Cap'n Proto installed, it may be out of date or is not being detected. Downloading and building libcapnp may take a while.
    already have c:\users\ivans_~1\appdata\local\temp\pip-build-7beli4\pycapnp\bundled\capnproto-c++
    error: [Error 2] El sistema no puede encontrar el archivo especificado

    ----------------------------------------
Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\ivans_~1\\appdata\\local\\temp\\pip-build-7beli4\\pycapnp\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\ivans_~1\appdata\local\temp\pip-usnlti-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\ivans_~1\appdata\local\temp\pip-build-7beli4\pycapnp\

I guess that the important line is the one that says:

c:\users\ivans_~1\appdata\local\temp\tmpojkh0v\vers.cpp(4) : fatal error C1083: Cannot open include file: 'capnp/common.h': No such file or directory

I get the same error when I try:

pip install pycapnp

Or even if I get pycapnp from https://github.com/jparyani/pycapnp.git and do:

pip install .

I suspect that the problem comes from a bad installation of the C++ Cap’n Proto Library (http://jparyani.github.io/pycapnp/install.html#c-cap-n-proto-library, which leads me to https://capnproto.org/install.html). I downloaded https://capnproto.org/capnproto-c++-win32-0.6.1.zip, and I can see that the missing file (common.h) is there, inside the capnproto-c+±0.6.1\src\capnp folder.

I imagine I have to copy those files somewhere, but I don’t know where.

Any ideas?

Thanks!

10

Are you sure you have C++ 11?

1 Like
11

Well, I downloaded the Microsoft Visual C++ Compiler for Python 2.7, is there anything else I need to do?

Thanks!

12

It looks like you’re on Windows. The pycapnp library is currently excluded from Windows builds. But more importantly, you probably want to install a binary package instead of compiling from source. You should be able to use pip to install a binary nupic package without the need for the nupic repo at all with the command pip install nupic.bindings. You can find plenty of explanations for using pip on Windows from google.

If you need to build from source, we currently have a very specific Windows build set up using MinGW that you can find here:

There is a lot of complexity there that you can ignore and it may be easier to look at the build output to see what exactly is being run:
https://ci.appveyor.com/project/numenta-ci/nupic-core/branch/master/job/tyancgaj5a1eppc2

Again, there’s a lot going on that can be ignored. The output is targeted towards developers so there is a lot of information that might not be relevant to you.

1 Like
13

The original problem was that I would like to save the model on disk. And as far as I know, the only non deprecated method was writeToFile.

However (I might be wrong, though), that method is only available from version 1.0.3 of nupic.bindings. If I install nupic and nupic.bindings using pip, I can get these versions:

nupic 1.0.3
nupic.bindings 1.0.2

Which is the recommended way of saving the model to file using those library versions?

Thanks a lot!

14

I see. On Windows, you should continue to use the “deprecated” method. See the docs here:
http://nupic.docs.numenta.org/stable/guides/serialization.html#deprecated-serialization-methods

1 Like
15

Sorry that I still need more help. As I cannot install pycapnp (because of that common.h file missing), I guess my only option is using cPickle. I don’t know what those “OPF Save/Load” and “Network Save/Load” are, but I don’t see any SpatialPooler instance there.

The Pickling method states that I need to save the C state, but I don’t know if that’s something implicitly done in the code shown, which just pickles and unpickles the SpatialPooler object. Do I have to do anything else in order to save that C state?

Besides, I see that the SpatialPooler class contains the methods save, write, load and read. Shouldn’t I use those?

I’m sorry, but I’m new to nupic, and all this is a bit confusing :worried:

Thanks!

16

Some C++ implementations have separate functions for writing the C state but the SpatialPooler implementations include all state (even C state) when pickling. So simply pickling and unpickling should be sufficient.

1 Like