Community Fork - CMake Enhancements and Package Management

I’ll update this comment as this discussion continues.

Tracking this topic in a new thread.

Goals
  • Update package management solution for nupic using package managers.
  • Reduce complexity of CMake scripts.
Package Managers
  • VCPKG (Windows)
  • Apt-Get (Unix)
Questions
  1. How does will this package management solution work for other, internal nupic packages?
  2. Can we make the VCPKG installation process a bit more user-friendly?
  3. How is this integrated with AppVeyor?

I created a small repository for proof-of-concept.
Look at any existing GitHub issues, if addressing any.

2 Likes

This would be great to have!
A way how this could be done comes to my mind:

  1. have 2 cmake files: “build-deps.cmake”, nupic.cmake
  2. for OSX/Linux (have built-in package managers) skip the build-deps (actual compiling from source), instead implement steps to download &install binary deps (ie apt install python3-dev)
  3. this can be tested in CI
  4. optionally, switch to VCPKG on Win, allowing us to drop the build-deps and remove all the dependency source codes from repo

Thoughts on Conan? Cross-platform C++ package management?

All open source? Couldn’t even get boost to install…

VCKPG has no version selection - there is a workaround, but it’s a pain. I think we may have to keep the distributed binaries.

Boost doesn’t have CMake support, but VCPKG ports it and includes the findboost.cmake, which has to be updated to support the latest boost.

I don’t understand if you suggest conan / vcpkg - I have no experience with either, so whatever works(?)
But there seems to be a problem with boost, if it’s the only problem, we include the headers only

Conan packages are all open source. There are hundreds of boost packages, all with different binaries. Other dependencies are the same. I was interested, since it’s cross-platform. We could use one package manager. I was wondering if anyone else had experience with it. After playing with it for a couple hours, I don’t think it will sole our problem well.

Vcpkg currently doesn’t have version control. Each version of vcpkg has a unique set of ports with a set version -
you have to install a certain version on vcpkg to get boost 1.65.1. CMake FindPackage breaks with boost 1.66.0, which is included in the latest vcpkg. I think vcpkg would be a good option for Windows, once it’s out of preview and has version control.

I’ll keep looking at more options.

1 Like

We could use CMake file(DOWNLOAD url file) to pull binaries from a different repository - nupic-packages.

https://cmake.org/cmake/help/v3.0/command/file.html

/boost
  /1.66
    x64-linux-gcc.tar.gz
    x64-windows-gcc.zip
    x64-windows-msvc.zip
    x86-linux-gcc.tar.gz
    x86-windows-gcc.zip
    x86-windows-msvc.zip

In a CMake function:

file(DOWNLOAD "https://github.com/numenta/nupic-packages/blob/master/${PACKAGE_NAME}/${PACKAGE_VERSION}/${PACKAGE_TRIPLET}" "${PACKAGE_NAME}.${PACKAGE_EXTENSION}")

Basically, a package manager lite that we control. For Windows, we could do the vcpkg craziness to get the correct version, then export it to the repository. For Linux… would apt-get just work? Would we need to patch anything?

1 Like

Good research, thanks!
I’m not happy with the results either.

This dependency situation is getting more nasty… :frowning:

I’m inclining to say, for linux/osx use apt/brew install boost-headers1.58-dev ,…
and for Win we ship the binaries and unpack (ideally not build) them in the repo, the cmake’s file(DOWNLOAD …) is being used already in the repo, see external/*.cmake

…?

Correct, wouldn’t we want to manage targets the binaries of all supported targets, albeit in a separate repository? New support could come in the form of a PR. That way, anyone on apple, unix, windows, etc. gets the same repositories, patches and everything. Like Conan’s approach, only nupic would be the user (e.g., ‘boost/1.66@nupic/stable’). CMake can then be conigured to download those binaries into the bin, based on the target triplet, which is pretty much nupic’s, file structure.

I’ll check it out… I thought we might want to just keep those in a separate repository.

@breznak S

Sounds good to me. Anything that gets us going should be prioritized. We can also fix later.

1 Like

+1 for that being in a separate repo (/just some cloud storage,…)
? not sure if there wouldn’t be conflicts , eg system boost vs ours. But the idea of binary deps for all targets and universal linking to that in Cmake is imho sound.

We had EXACTLY this approach down in the past, with nupic-platform-linux64,... @rhyolight do you recall why did we abandon it? Were there any caveats?

We abandoned those dependency repos because CI environments got more stable and we were able to declare more and more dependencies naturally as the project matured. At some point we stopped needing them.

@heilerm what if we stick to that too? Unix have typically good package capabilities, for Win we make the binary-deps repo as you suggest. If we see problems with either approach, we can shift to one, or the other solution?

1 Like

It honestly always felt like the wrong way to do it.

@rhyolight What would have been the better way?

When you add binaries to a repo, you’re making assumptions about the environment in which it can be run, therefore you’re also putting possible restrictions on it as well.

Best IMO would be not to have any binaries in any repositories. This may be unrealistic, but it is something to shoot for.

1 Like

I think vcpkg will get there. We can even use it to build and export the dependencies we need, dealing with the versioning issue. Once it gets there, zero managed binaries. We will need some sort of manifest, though.

I updated my project with a few enhancements:

  • Each project has its own file structure.
  • The core header files filter is a bit flatter.
  • The Python binding project has an example.
  • The CLR C# project has an example.

I’d like to test this on a Linux box…

The C#, Python, and future projects might need a template in order to hook things up correctly, given any changes to CMake. But… do they even belong in this repository?

Still TODO:

  • Install
  • Tests

If you think any of this is wrong, or you want to try something else, let me know.

A can try to test! Or help you setup Travis for Linux testing.