Proposal to introduce pybind for move toward Python 3 compatibility


#41

This one? https://github.com/chhenning/nupic

Correct. Please note that this repo not only contains nupic.core but also nupic. The purpose is the be self contained with all dependencies, like boost, pybind, etc. Ideally no outside code.

You guessed right, it’s not a fork from Numenta but a repo from scratch.


#42

That’s why it wouldn’t merge :stuck_out_tongue:

Thing is you’re making it very difficult for the community to participate with existing/prior code. This would mean PRs from other forks (numenta/nupic.core has 195), review future code-drops from Numenta, and for numenta to integrate yours, for 3rd party projects built on nupic…

I’d suggest early on the beginning: try to rebase it on nupic.core (+nupic, if that’s possible)?

Something like:

  • fork nupic.core
  • pull in all of nupic
  • drastic cleanup, reshuffle files as you wish (all with git commands!)
  • reapply your changes

#43

This sounds reasonable to me. How would I fork nupic into a forked nupic.core?


#44

I’ll have to check if you can then make PRs ALSO from the other repo.
But I’ll be an improvement to the current state if you rebase on numenta/nupic.core. (I think we can assume most code will be from this repo?)

Option B is using git submodules, it’s like simlink to the other repo. But then you cannot shuffle files between the “name-spaces”. If you could have something like:

  • C
    • C/here is PY/
  • other c++

that would work ok.


#45

OK, great idea! Let me sleep over it and I’ll give it a try over the weekend.


#46

OK, here is my understanding. We would like to keep the history of Numenta’s nupic.core repo into the community repo. The easiest way to accomplish that would be to fork and then do all the necessary transformations (read, file operations and code changes) on such fork. Correct?
The reason to so is that we can merge code changes from Numenta but also the reverse if wanted.

My current design incorporates also Numenta’s nupic repo into the same community fork repo. I’m not hard set on this and willing to change back to the original repo structure. For that, I’ll fork from Numenta’s nupic and apply my changes python 3 changes. I don’t think we need the python 2.7 code, correct? Full steam ahead, right? :wink:

The only question that’s left is where to house my Visual Studio solution. I like to have all code in one solution (read nupic.core and nupic). Should I do a separate repo for that then? I assume a cmake script would then combine the two repo and create a VS solution?

Does that all make sense?


#47

@chhenning This sounds good to me.

Although, having the nupic in the same repro with nupic.core would complicate things a little for me. For example, for those wanting a C# environment may not even have Python installed. I think we should try to keep nupic Python and nupic.core in separate repositories (i.e. solutions). It will be hard enough to construct the nupic.core such that a Python user does not need C# installed and C# user is not required to have Python installed.

Obviously for someone using the nupic Python only code would need Python. I am assuming that there will be a similar repository for C# only code that parallels the code in the nupic Python repository. These would contain examples and helper routines that someone could use to construct their projects. These would be calling into nupic.core for the main parts (or at least those parts of the core that they need).

Let’s think about who is using which code… People like us would be changing the core stuff but most people will just be users of the resulting library and would most likely only be interested in the nupic Python or nupic C# for code that they would modify to make their own experiments. These people would just build the core once to get the libraries and then go to the other repositories. Windows people could even use pre-built nupic.core libraries and never need to download the full nupic.core repository.

This could then also be extended to any other language we might want to support.

What do you think?


#48

The easiest way to accomplish that would be to fork and then do all the necessary transformations (read, file operations and code changes) on such fork. Correct?

Yes, that way will allow easy community integration with your repo based on Numenta’s.

I don’t think we need the python 2.7 code, correct? Full steam ahead, right?

+1 to dropping py2.7

I’m not hard set on this and willing to change back to the original repo structure. For that, I’ll fork from Numenta’s nupic and apply my changes python 3 changes.

++1. That would be awesome. Let’s keep the things a little bit in order, develop your changes on it. And if later it proves beneficial to merge, it’s always possible.

The only question that’s left is where to house my Visual Studio solution. I like to have all code in one solution (read nupic.core and nupic). Should I do a separate repo for that then? I assume a cmake script would then combine the two repo and create a VS solution?

Do you want this as a public endeavor, or just for private use? I’d go with cmake delegating to the two cmakes below. But not sure how you’d handle commits to this merged repo…


#49

@breznak OK, I’ll start this weekend. When I fork I’ll create a branch which contains the original Numenta stuff and also create a new branch “develop” where I’ll keep my changes.

Does that sound reasonable?


#50

@David_Keeney I agree with your statements!


#51

Yes, you just git checkout -b py3 from the numenta/nupic.core/master and then your changes can be applied easily.

I’m getting to the point of having CI running this weekend.


#52

@chhenning When you get your changes in I will replace the CMake’s with ones that build the Python interface parts separate from the rest of the core. What I have will need to be changed since I know you removed some stuff and I expect some files will be in different places.

Now…if we move directories around or rename them, what will happen when we try to get a change in numenta’s repository into our repository?


#53

@breznak Wouldn’t “community fork” be a better branch name? For nupic.core the changes are in regards to:

  • removing python c API calls
  • removing python helper classes
  • removing unnecessary dependencies like APR
  • removing cap’n proto
  • cleaning up some compiler warnings
  • cleaning up code for MS compiler
  • etc.

In essence, it’s not just python related.


#54

I see. “community fork” does not say much about its contents either. So “use_pybind”?
I guess we’ll better discuss that on the code, but so far the only breaking change without similar functionality will the the removal of capnp, right?


#55

That’s the nice thing, git will handle that for both cases (you must be using the git mv/rm commands)
Ofc for removed files it cannot resolve the problem, so will ask you what to do (keep new/ keep removed)


#56

@breznak Correct. Although, there is some work to be done in the os layer (folder). For instance I recently removed all platform dependent code for implementing a timer (stopwatch) with std::chrono. There are some holes to be filled with things like boost::filesystem and boost::dll (for loading dynamic libs). Nothing very complicated.


#57

Perhaps name the fork ‘community.core’ or even 'nupic.community.core’
Since this is the stuff that is common for all languages plus the language interfaces.

For all of the Python stuff (formally in nupic) name it ‘nupic.community.py’.
The corresponding C# stuff (something parallel to what is in nupic) could be ‘nupic.community.cs’

I don’t know… Does anyone else have ideas?


#58

I thought it’s about feature branch, not a repo name. If the latter, do you want it to the one already in htm-community org? https://discourse.numenta.org/t/new-htm-community-forks-nupic-cpp-nupic-py/
I’d definitely like to see the Py3, boost, cleanup, pybind there.
I can add you both to maintainers of the repo.


#59

Oh, sorry. I did not read it close enough. I am still grumbling about the name of the repo.


#60

I am still grumbling about the name of the repo.

Cool, make a thread and let’s finish it there? nupic.corelibs? …??