Nupic.core Extraction Plan

Engineering NuPIC
,
1

NuPIC is currently undergoing a major refactoring, with the goal of extracting a C++ “core” that contains all CLA algorithms. This will provide a core repository so other language bindings can be created aside from Python. The extraction plan for this refactoring is detailed below and kept up-to-date by Matt Taylor.

NuPIC Today

NuPIC today

  • One repo: nupic
  • primary interface: python
    • all examples are python
    • all tutorials are python
  • underlying C++ interface that no one uses
  • C++ is compiled with SWIG with python bindings

The Goal

NuPIC today
nupic-core-extraction-goal.jpg710×431 44.5 KB

  • nupic.core contains all C++
    • all algorithms are implemented in C++
    • complete API documentation
    • tutorials for direct usage of C++ API with C++ client
    • tutorial for creating language bindings
  • python binding and client created from nupic project python code
    • community is responsible for creating bindings and clients in other environments
  • undecided whether language bindings and clients will be combined or in different repos
  • undecided naming scheme for bindings and clients, but we should provide guidance with python

Step 1: Initial Split

Summary of initiatives:

  1. Split nupic into two repositories
  • nupic.core
  • nupic
  1. Update nupic build process to accommodate these changes
  • checkout and build nupic.core as a dependency
  • decide what a passing build means by these new standards
  1. Establish an independent nupic.core build process
  • have nupic start using the nupic.core build instead of building it itself

NuPIC today

nupic.core will contain only C++ code with minimal dependencies. nupic will contain everything else currently within nupic.

Tasks:

  • Clean up existing build script nupic #583
  • Remove all python traces from nupic in preparation for nupic.core nupic #584
  • Create nupic.core repository in github nupic #585
  • Remove C++ code from nupic repository and add nupic.core dependency nupic #587
  • Ensure nupic python tests pass after nupic.core dependency has been added nupic #588
  • Create a mechanism to update the nupic.core dependency from nupic with every push to nupic.core master #590
  • Update nupic build to CMake nupic #647
    • Travis runs cmake builds in clang & gcc
    • Update README build instructions for cmake
    • Deprecates build.sh process in favor of cmake
  • Remove old build process and artifacts from nupic nupic #673
  • Establish independent nupic.core build process nupic.core #4
  • Switch nupic to use the nupic.core build system nupic #625

Step 2: Prepare for nupic.core release

In order to provide a stable nupic.core for developers to program language bindings and clients against, we need to have a stable v1.0 release. Requirements for a stable release are defined below.

Requirements

The following requirements do not affect each other and can be worked upon simultaneously.

1. Define API with Tests

To ensure backwards-compatibility, a new test suite must be created for nupic.core that exercises all public API endpoints. These tests do not need to be extensive functional tests, but they do need to validate the signatures of the endpoints (both input and output) and assert basic expectations about the structure of the public API. This test suite MUST cover all classes and functions that are expected to be a part of the API.

NuPIC today

The API tests exist to establish a contract that can be run in the CI pipeline that will identify breaking API changes in future pull requests. It also makes it obvious for reviewers when a PR modifies an API tests that backwards compatibility is intentionally being broken.

2. Consolidate All CLA Algorithms in nupic.core

nupic.core MUST contain all CLA algorithms before it can be released, simply because it is incomplete otherwise.

NuPIC today

The following necessary components must be translated from Python in nupic into C++ in nupic.core:

  • current “temporal pooler” (sequence memory)

This should not change the nupic.core API defined by the API tests.

3. Remove Unused Code From nupic.core

Identify and remove unused code.

Step 3: Release nupic.core-v1.0

Creating a stable release fork for nupic.core is essential for client programmers to create tools for running NuPIC in different environments.

We’ll discuss options for the mechanics of this work on the nupic-hackers mailing list at the appropriate time and fill this section in with the results.

Step 4: Create python language binding and client library

Our first official client library will be a conversion of nupic into nupic-py (bindings) and nupic-py-client (OPF client). This will be a functional example of how to create a client for other programmers of bindings and clients.

Work on this step can be done immediately after Step 1 above is complete. Steps 2/3 above are not prerequisites.

Summary of initiatives:

  1. Split nupic into two repositories:
  • nupic-py: contains only Python language bindings
  • nupic-py-client: contains OPF, encoders, etc. used by Grok
  1. Update CI process to accommodate three repositories
  • Does a nupic.core change trigger builds for all repos synchronously?
  • How does the bindings repo include the core?
  • How does the python client include the bindings?

NuPIC today

2

Is this inline with the htm.java roadmap? Is htm.java going to stay a port? C# is my current wheelhouse… I was investigating whether or not I could bind/wrap nupic.core with it, but it seems not all has moved there and its in maintenance mode, of course.

3

This plan has nothing to do with htm.java. It is going to stay a port. I think you can create C# bindings for nupic.core using swig.

4

Cool. That’s what I was hoping. I was planning on using Swig… I’ll keep going, although I see @chhenning is working on a VS 2017 branch… might wait for that.

1 Like