# Re-Organisation of the C++, PY Community repos

**URL:** <https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327>\
**Category:** NuPIC Community Fork\
**Tags:** community, structure, repo, hackers\
**Created:** [January 18, 2018, 11:15pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327 "2018-01-18T23:15:59Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![chhenning](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/chhenning/32/3821_2.png) [@chhenning](https://discourse.numenta.org/u/chhenning)\
**Post date:** [January 19, 2018, 3:28am UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/21 "2018-01-19T03:28:35Z")

</div>

> [@breznak](#):
>
> These are binaries, folder structure, right? Not separate repositories?

Correct.

> [@breznak](#):
>
> And these nupic.bindings.py(2) :

In a nupic installation the bindings are actually a mixture of python and cpp modules (algorithm, engine, math)

> [@breznak](#):
>
> yaml-cpp - static cpp yaml lib
> 
> even removed?, serialization.

As far as I can see it’s just used for configuration.

> [@breznak](#):
>
> I can see the benefits of merged py+cpp repos.
> 
> TL;DR: we more or less agree on the structure, just should consider STRICT isolation of separate layers. And possible modularization in repos: What will happen when @David_Keeney merges the c# bindings? And some others? And alternative implementations of the algorithms? …I think the repo might get huge, too crowded with issues and activities to manage.
> 
> Another thing is 3rd party projects that would like to use some functionality (swarming, python, c++ iib) and don’t want to pull,manage,build all the other stuff.

I think for now we are good. In the future we might restructure. I’m still stuck in “small” issues like recompiling nupic.core with 64bit float vs 32floats. Or even 128 floats…

Hope you don’t take it the wrong way. Right now I really like having everything in one visual studio solution, including nupic. Great for testing and tinkering.  
cmake is on the todo list!

> [@breznak](#):
>
> Or we could brainstorm the suggested alternative which provides way for the both worlds:

Yes, let’s brainstorm!

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 19, 2018, 3:44am UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/22 "2018-01-19T03:44:02Z")

</div>

> [@chhenning](#):
>
> cmake is on the todo list!

I’m eager to try this on linux

---

<div class="post-metadata">

**Author:** ![thanh-binh.to](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/thanh-binh.to/32/1793_2.png) [@thanh-binh.to](https://discourse.numenta.org/u/thanh-binh.to)\
**Post date:** [January 19, 2018, 6:03am UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/23 "2018-01-19T06:03:11Z")

</div>

@David_Keeney we are using C++ Network API and Region for working with different hierarchies in htmresearch

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 19, 2018, 12:51pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/24 "2018-01-19T12:51:11Z")

</div>

I am looking for people (C++, Py,…) who would like to participate in the community Reviewers team. I think it’s important to see the code with more eyes (not only for bugs, but for design, usability). But I also hope for quick feedback and rapid developement in the community repo.

- I’ve set “rules” for merging that at least one review is required to get the new code in.

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 19, 2018, 1:15pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/25 "2018-01-19T13:15:55Z")

</div>

## Goals

Q: What would you like the goals, **direction of a community repo** would be?  
After we get more ideas, I’ll make a poll again.

For me it is:

- at start, continue with active depelopment based on numenta/nupic.core/py
- active PR review, fixes, open for big changes,…
- keep multiplatform support
- keep high level of code (tests, reviews)
- gradual development to new big features (optimization, py3, restructuring,…)
- ease of use for the programmers,developers(=us 😉 )

---

<div class="post-metadata">

**Author:** ![David\_Keeney](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/david_keeney/32/1616_2.png) [@David\_Keeney](https://discourse.numenta.org/u/David_Keeney)\
**Post date:** [January 19, 2018, 4:00pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/26 "2018-01-19T16:00:55Z")

</div>

All of this discussion is good. But we must keep in mind what this library is for. This is not a product or production application. This is a framework for experimentation.

From my viewpoint this library is intended for people who want to experiment with their own implementations of any of the components.

- So the first priority is understandability of how the algorithms in the library work.
- The second priority is flexibility in how the parts can be connected.
- The third priority is flexibility on platform and programming language of the user.
- Performance is important but not at the expense of the other three goals.

We should assume that the number of algorithms and variations on those algorithms will continue to grow as people find new things that work…after all that is the purpose of the library. Consequently do not expect any of the API to remain constant. As new ideas are presented we should try to incorporate them. For example, if someone discovers a new way to do SP and has a working implementation in Python we should help them port that to C++ and add it to the library if they are unable to do that themselves. I would not expect to see polished code being offered and that should be ok. Every module in the library must have a corresponding unit test module but don’t expect offered modules to have one so we must help them provide one.

@breznak your thing seems to be C++ optimization…that is great. You can help optimize the submitted modules as long as it does not make it harder to understand or loose flexibility.

Having said that, the actual layout of the library should be focused on how easy it is to understand even by someone that is not a professional programmer. In my opinion it is not mandatory for the community core library to be a clone of nupic’s production core as long as we can identify the changes they have made so we can incorporate them into our library. I expect there to be considerable deviation.

---

<div class="post-metadata">

**Author:** ![David\_Keeney](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/david_keeney/32/1616_2.png) [@David\_Keeney](https://discourse.numenta.org/u/David_Keeney)\
**Post date:** [January 19, 2018, 4:02pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/27 "2018-01-19T16:02:24Z")

</div>

@thanh-binh.to Thank you. That is reason enough to include it in the C# interface.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 19, 2018, 4:27pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/28 "2018-01-19T16:27:32Z")

</div>

> [@David\_Keeney](#):
>
> @rhyolight do you know if the Network ‘framework’ is being used?

We use it heavily internally. The OPF is built on top of it. I suggest you keep it, it provides the flexibility to construct layers and columns.

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 19, 2018, 9:54pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/29 "2018-01-19T21:54:58Z")

</div>

Thank you for your points, will def. add them to the poll.

> This is not a product or production application. This is a framework for experimentation.

I agree, in a way. I’m open for more rapid and extreme changes, but on the other hand, I’d like to have a fork that is “an actively developed continuation of the Numenta’s repositories”. So that Numenta can try to sync once in a while, if they wish, and people who build their apps on top of it can continue to use a fixed and developed descendant. So I’ll also add

- compatibility (more or less) with the current Numenta API
- rapid (vs conservative) development (API breakage)
- unit-test coverage (each new feature is tested)
- keep c++ / Py feature+API parity (vs the repos can separately diverge and live on their own)

Note, I’m collecting ideas to ask here, not that I’d agree with all the points I’m listing here.

> library should be focused on how easy it is to understand even by someone that is not a professional programmer

I’m not sure about this one. Either they are scientists and focus mainly on the papers/NeuroSci, or programmers who focus (and know) the internal workings, or application users, who use products based on HTM (Grok, HTM schools, …)…imho

I would not really care about whitespaces and coding-style so much (always Matt had to punch me to do that 😉 )

> …if someone discovers a new way to do SP and has a working implementation in Python we should help them port that to C++ and add it to the library if they are unable to do that themselves

Careful with this, of course we’ll do it if we like that or it’s uber cool feature; but you might soon end up porting code you are not interested at.

More ponts, ideas for the poll, what you want from future nupic?

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 19, 2018, 9:57pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/30 "2018-01-19T21:57:33Z")

</div>

- pure python functionality (no other dependency on bindings, for quick prototyping)
- focus on Py repo
- focus on c++ repo
- provide releases for binary installs (pypi)

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 20, 2018, 3:58pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/31 "2018-01-20T15:58:56Z")

</div>

Great job coordinating, you all. It would be a Very Good Thing to get everyone working on the same forked codebase with a set of objectives. You seem to be doing in the right direction.

> [@David\_Keeney](#):
>
> For example, if someone discovers a new way to do SP and has a working implementation in Python we should help them port that to C++ and add it to the library

Just be careful about letting “new algorithms” into the project. When this happens, be very clear about where they originated, whether they are biologically inspired or not (cite papers). It will help in the future.

---

<div class="post-metadata">

**Author:** ![David\_Keeney](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/david_keeney/32/1616_2.png) [@David\_Keeney](https://discourse.numenta.org/u/David_Keeney)\
**Post date:** [January 20, 2018, 7:47pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/32 "2018-01-20T19:47:06Z")

</div>

@rhyolight Ah, yes I agree. These should be HTM algorithms. I was thinking in terms of some of the variations of the HTM modules listed in the API specifications…like backtracking TM, and perhaps some more encoders and classifiers or even some monitoring tools. Hopefully some new things will eventually come out of Numenta’s current research that we can add.

---

<div class="post-metadata">

**Author:** ![heilerm](https://avatars.discourse-cdn.com/v4/letter/h/9fc29f/32.png) [@heilerm](https://discourse.numenta.org/u/heilerm)\
**Post date:** [January 25, 2018, 7:11pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/33 "2018-01-25T19:11:42Z")

</div>

I know you already set up the repositories, but…  
My stab at it…

```auto
// core repository (C++)
/nupic.core
  /packages // [machine]/[vendor]/[operating system]/[package].[tar.gz|zip]
    /x64
      /gcc
        /linux (*.tar.gz)
          ...
        /windows (*.zip)
          ...
      /msvc
        /windows (*.zip)
          ...
    /x86
      /gcc
        /linux (*.tar.gz)
          ...
        /windows (*.zip)
          ...
      /msvc
        /windows (*.zip)
          ...
  /modules // can these go in separate repositories?
    /nupic.core
      /include
        /nupic // headers
          ...
      /src
        ...
    /nupic.core.cs // bindings
      /include
        /nupic // headers
          ...
      /src // header-only?
        ...
    /nupic.core.py // bindings
      /include
        /nupic // headers
          ...
      /src // header-only?
        ...
  /tests
    ...
CMakeLists.txt

// client repository (C++)
/nupic -> [nupic.core]
  /include
    /nupic // headers
      ...
  /src
    ...

// client repository (C#)
/nupic.cs -> [nupic.core, nupic.core.cs]
  ...

// client repository (Python)
/nupic.py -> [nupic.core, nupic.core.py]
  ...

```

---

<div class="post-metadata">

**Author:** ![breznak](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/breznak/32/1935_2.png) [@breznak](https://discourse.numenta.org/u/breznak)\
**Post date:** [January 25, 2018, 8:56pm UTC](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327/34 "2018-01-25T20:56:08Z")

</div>

the reorganization did not happen yet. There’s been a discussion with good points for both sides.  
So far the repo tries to keep things with minimal changes and gradually evolve.

The related topics seem to be repo structure & how to manage dependencies.

> ```
> /gcc
> 
> ```

```
    /linux (*.tar.gz)
      ...

```

instead of stuffing the binaries into the repo, can we just point to a public release and use the cmake: `file(DOWNLOAD ..) ` as you’ve suggested somewhere else?

What is again difference between a nupic.core(c++) and nupic.core.client(c++) ?

Seeing the problems we ran into in the Py3 PR, I’d like to ask What are the real benefits this brings us? Compared to the status quo. As each of these changes is quite a lot of work.

[Previous page](https://discourse.numenta.org/t/re-organisation-of-the-c-py-community-repos/3327.md?page=1)