Should we make NuPIC python 3 compliant?

I’m using NuPIC to learn Python, so I’ll take a crack at it. Should be a good way to help solidify my understanding of the language. Having a newb to a language to port code is probably not the best idea though :blush:. Could be at least a good starting point and have some seasoned Python experts come in and clean it up afterwards.

I can give a look if you are in trouble !

1 Like

I don’t use NuPIC, but my two cents as someone who does a very large amount of python programming: python3 compliance (i.e. making sure the code runs on python3) is a good idea. But requiring features unique to python3 is not. The compatibility differences between the two versions are the biggest source of friction in my workflow, when I’m integrating other machine learning researchers’ code with my own. So my opinion on these questions is that where at all possible (which is everywhere), write code that works in both versions of the language.

4 Likes

In case you needed more motivation to migrate to python3, numpy (used heavily by nupic) is losing py2 support in 2020:

1 Like

I’m exploring the possibility to move all python code to boost::python. I’m in contact with one of the developers and he told me that boost python supports version 2.7 and 3.x.

There even is a numpy extension:
http://www.boost.org/doc/libs/1_65_1/libs/python/doc/html/numpy/index.html

Does anyone have a good reason not to do that?

A post was split to a new topic: Proposal to introduce pybind for move toward Python 3 compatibility

Hey guys, sorry I’m 10 months late.

Is there someone still interested in making nupic compatible with python3? As a long-time python programmer and as systems dev, I highly recommend a python3 compat nupic. Here are my high-level whys in my opinion.

  1. Python 2 EOL is fast approaching. As we get closer to this, nupic’s development appeal will gradually decrease. Almost all my project as of this stage are mostly python 3 and most python devs love simplicity, that is why as much as possible we would prefer working only for one interpreter, if that is possible of course.

  2. The future of python is python 3. Some say it is best to support both python 2 and 3, however, I believe that this only applies to “uncontrollable” scenarios. For nupic, I believe that the friction and breakages that may be brought by the transition (python 2 to 3) will be less heavy than the benefits that it will bring.

  3. I do not know the details of the code as of this moment, however, nupic may benefit very much from python’s asyncio libraries.

Let me know if there is someone who’s keen to port nupic to python3 as I would be keen to help. I will take this as a challenge with my ulterior motive that is to also learn the internals of nupic, and hopefully understand HTM better.

2 Likes

I’m also still interested in helping with this effort. My Python skills are not up to par for tackling it myself, unfortunately.

3 Likes

My opinion, just my opinion of course, python is easy to learn, probably the easiest of all especially if the simplest path is taken during coding. My experience tells me that python will help you learn the code or logic more than learning the language itself. So it allows non-python programmers to understand python code in an effective way.

Thanks for the interest, now where should we start? :grinning: I hope there will be more people who’d join the party. For now, I will have a look at the nupic code and probably come up with a high-level conversion strategy (python 2 to 3).

2 Likes

Have you started working on this already? Let me know, keen to help. One of the things we need to know is that whether the conversion we will be pursuing is one of the following paths;

  1. python2.7 to python3.x
  2. python2.7 to python3.x (supports python2.7)
  3. python2.7 to python3.x (supports python2.7 and older versions)

#1 would be the easiest path, however in reality many ML practitioners out there still use 2.7 so supporting 2.7 would be more beneficial to everyone at least as of this stage. What are your thoughts?

1 Like

Hi Jose,

I worked on this a few months ago. The code here works with python 3. There are still a few tests that are failing and it’s mostly due to behavioral differences between python 2.7 and python 3. Have a look here here.

Hope this helps.

Christian

2 Likes

Hi @chhenning,

Awesome! Great work! I’ll have a look.

I noticed you’ve opened some issues, are these the issues you’ve encountered during conversion? By the way, what is your method of converting python 2 to 3? Will your changes support both 2 or 3 or just 3?

Cheers

1 Like

I did use the 2to3.py tool. Which generated python 3 code.

Here is the output

There are ways to have python code that works with python 2 AND python 3 but I think that’s not what we want.

1 Like

We are working on a new HTM-Community version of the C++ core (nupic.cpp) to go with that python 3. https://github.com/htm-community/nupic.cpp

Unfortunately it is not ready for prime time yet. We still need to replace SWIG with pybind11 and a few other goodies. (see Issues). Progress has been slow.

2 Likes

Great!

The information so far leaves me some questions, sorry I’m new to the HTM community. Could you guys please share your inputs when possible?

  1. What does nupic community version mean? In terms of long-term goals and ownership, how does it differ from the nupic code provided by Numenta?

  2. What is the need for building a nupic community version? Is Numenta’s nupic unmanageable or reached its EOL?

  3. Is there a consolidated effort for updating nupic & nupic.core to python 3?

  4. Is nupic.core conversion purely a conversion effort or an upgrade as well?

I just like to catch up with what’s done so far.

Cheers

1 Like

Jose, these are great questions and we need to re-visit these periodically to make sure we are keeping on track.

  1. What does nupic community version mean? In terms of long-term goals and ownership, how does it differ from the nupic code provided by Numenta?

The nupic community version is a fork off of the Numenta nupic code set and is maintained entirely by community members. It remains under the Numenta open source license. The goal is to remain true to the HTM algorithms developed by Numenta as a foundation and provide a framework for the community to experiment with their own ideas and concepts without having to start from scratch.

What is the need for building a nupic community version? Is Numenta’s nupic unmanageable or reached its EOL?

The Numenta code base has been somewhat locked in an older technology with Python 2.7 and older C++ libraries. The Numenta staff have been focusing their attention on extending the model where it is most important rather than expending resources on upgrading to the latest tools. The older technology is probably not a problem for the Numenta staff but for the community members trying to learn the HTM model it is a distraction if not a deterrent.

  1. Is there a consolidated effort for updating nupic & nupic.core to python 3?
  2. Is nupic.core conversion purely a conversion effort or an upgrade as well?

The objective is to upgrade the nupic Python code to python 3 and reduce the complications of installing and learning the code for community members. The first effort is to just mechanically convert the Python code to python 3 using the conversion tools. If more community members are interested they can take on clean up and streamlining portions of the Python code as well.

My personal effort has been on the C++ library. The C++ portion of the code was originally intended to be just an extension of the Python code base to provide a little more performance. With the community version, the C++ portion is being flushed out to the point that it could also be a stand-alone library that can be used by community members that may be interested in experimenting entirely in C++. We are removing older unused logic, upgrading to C++17, removing most of the old dependencies, and streamlining the code. It is intended that it could also be a foundation library for CSharp and possibly other languages. We are targeting Linux, OSX, and Windows platforms, both 64bit and 32bit. The guide is to retain the nupic HTM API as much as possible.

The first step was to remove the capnproto serialization from the C++ library and replace it with simple binary streaming serialization. That step is nearly complete. This simplified the code base considerably. The next step is replacing the APR and boost dependencies with C++17 std:: library routines. Then we will replace SWIG with pybind11 and isolate the Python interface as an independent interface to the C++ library followed by an effort to streamline the infrastructure. Then we can port more Python modules to C++ as it seems appropriate.

As the Numenta staff expand the HTM model and firm up the logic we will extend the community C++ library to include these new algorithms.

Anyone interested in participating, take a look at the issues list in GitHub https://github.com/htm-community/nupic.cpp and join in the conversation. If you find something you would like to do, create a fork and make it happen.

3 Likes

I definitely support the community version efforts, so don’t take this comment the wrong way. I personally believe Python 3 compatibility in upstream NuPIC is necessary, and that it should be made to work with both Python 2 and Python 3. The reason is because there are still many other libraries which are not yet Python 3 compatible, so having compatibility for both provides the best opportunity for folks to include NuPIC in their projects. At some point down the road when Python 2 is past EOL and majority of legacy projects have themselves either been updated or EOL, then would be a better time to do a full refactor and take advantage of Python 3 specific improvements.

2 Likes

I agree that the ability to support both Python 2 and 3 is very desirable. We welcome any volunteers that would like to take on that project? You can start by adding an Issue to the nupic.py repository and describe what you would like to do. Perhaps starting with @chhenning 's contribution and testing for compatibility with both Python 2 and 3.

2 Likes

@Jose_Cueto are you interested in working this into upstream NuPIC or the community version? If the former, I’ll be happy to help where I can.

@David_Keeney Thank you very much for the overview it was extremely helpful for newcomers like me. Now I understand why there is a community version of nupic. Great work on the efforts done so far.

@Paul_Lamb I’d like to help on the conversion for sure. As of now, it is a bit hard to decide which code base I’d work on converting py2 to py3. My main concern is that I do not want our efforts to be duplicated. My assumption (maybe I’m wrong) is that these two codebases may have diverged already or will diverge in the future, so efforts made might not be beneficial to both codebases and impact would not be significant. What are your thoughts on this? At this stage, I’m still trying to have some time to read the code and understand its high-level architecture, this is sort of checking out what type of beast I’ll be dealing with for the conversion work. The code, by the way, is highly OO, I don’t have anything against it, it’s just that it requires a steeper learning curve than a loosely OO code. Another thing to determine is that for the upstream code if there is a need to convert the C++ code as well, because this will be a different full-time task besides the py code conversion. I will have a think about this.

1 Like