Community Fork - CMake Enhancements and Package Management

heilerm · January 24, 2018, 7:23pm

I’ll update this comment as this discussion continues.

Tracking this topic in a new thread.

Goals

Update package management solution for nupic using package managers.
Reduce complexity of CMake scripts.

Package Managers

VCPKG (Windows)
Apt-Get (Unix)

Questions

How does will this package management solution work for other, internal nupic packages?
Can we make the VCPKG installation process a bit more user-friendly?
How is this integrated with AppVeyor?

I created a small repository for proof-of-concept.
Look at any existing GitHub issues, if addressing any.

breznak · January 24, 2018, 7:52pm

This would be great to have!
A way how this could be done comes to my mind:

have 2 cmake files: “build-deps.cmake”, nupic.cmake
for OSX/Linux (have built-in package managers) skip the build-deps (actual compiling from source), instead implement steps to download &install binary deps (ie apt install python3-dev)
this can be tested in CI
optionally, switch to VCPKG on Win, allowing us to drop the build-deps and remove all the dependency source codes from repo

heilerm · January 25, 2018, 4:27am

Thoughts on Conan? Cross-platform C++ package management?

heilerm · January 25, 2018, 5:22am

All open source? Couldn’t even get boost to install…

heilerm · January 25, 2018, 6:06am

VCKPG has no version selection - there is a workaround, but it’s a pain. I think we may have to keep the distributed binaries.

Boost doesn’t have CMake support, but VCPKG ports it and includes the findboost.cmake, which has to be updated to support the latest boost.

breznak · January 25, 2018, 9:08am

I don’t understand if you suggest conan / vcpkg - I have no experience with either, so whatever works(?)
But there seems to be a problem with boost, if it’s the only problem, we include the headers only

heilerm · January 25, 2018, 1:56pm

Conan packages are all open source. There are hundreds of boost packages, all with different binaries. Other dependencies are the same. I was interested, since it’s cross-platform. We could use one package manager. I was wondering if anyone else had experience with it. After playing with it for a couple hours, I don’t think it will sole our problem well.

github.com

Microsoft/vcpkg/blob/master/docs/about/faq.md#why-not-conan

# Frequently Asked Questions

## Can I contribute a new library?
Yes! Start out by reading our [contribution guidelines](https://github.com/Microsoft/vcpkg/blob/master/CONTRIBUTING.md).

## Can Vcpkg create pre-built binary packages? What is the binary format used by Vcpkg?
Yes! See [the `export` command](../users/integration.md#export).

## How do I update libraries?
The `vcpkg update` command lists all packages which are out-of-sync with your current portfiles. To update a package, follow the instructions in the command.

## How do I get more libraries?
The list of libraries is enumerated from the [`ports\`](https://github.com/Microsoft/vcpkg/blob/master/ports) directory. By design, you can add and remove libraries from this directory as you see fit for yourself or your company -- see our examples on packaging [zipfiles](../examples/packaging-zipfiles.md) and [GitHub repos](../examples/packaging-github-repos.md).

We recommend cloning directly from [GitHub](https://github.com/microsoft/vcpkg) and using `git pull` to update the list of portfiles. Once you've updated your portfiles, `vcpkg update` will indicate any installed libraries that are now out of date.

## Can I build a private library with this tool?
Yes. Follow [our packaging zlib Example](../examples/packaging-zipfiles.md) for creating a portfile using a fake URL. Then, either pre-seed the `downloads\` folder with a zip containing your private sources or replace the normal calls to `vcpkg_download_distfile` and `vcpkg_extract_source_archive` with functions that unpack your source code.

## Can I use a prebuilt private library with this tool?

This file has been truncated. show original

Vcpkg currently doesn’t have version control. Each version of vcpkg has a unique set of ports with a set version -
you have to install a certain version on vcpkg to get boost 1.65.1. CMake FindPackage breaks with boost 1.66.0, which is included in the latest vcpkg. I think vcpkg would be a good option for Windows, once it’s out of preview and has version control.

I’ll keep looking at more options.

heilerm · January 25, 2018, 2:21pm

We could use CMake file(DOWNLOAD url file) to pull binaries from a different repository - nupic-packages.

https://cmake.org/cmake/help/v3.0/command/file.html

/boost
  /1.66
    x64-linux-gcc.tar.gz
    x64-windows-gcc.zip
    x64-windows-msvc.zip
    x86-linux-gcc.tar.gz
    x86-windows-gcc.zip
    x86-windows-msvc.zip

In a CMake function:

file(DOWNLOAD "https://github.com/numenta/nupic-packages/blob/master/${PACKAGE_NAME}/${PACKAGE_VERSION}/${PACKAGE_TRIPLET}" "${PACKAGE_NAME}.${PACKAGE_EXTENSION}")

Basically, a package manager lite that we control. For Windows, we could do the vcpkg craziness to get the correct version, then export it to the repository. For Linux… would apt-get just work? Would we need to patch anything?

breznak · January 25, 2018, 8:31pm

Good research, thanks!
I’m not happy with the results either.

This dependency situation is getting more nasty…

I’m inclining to say, for linux/osx use apt/brew install boost-headers1.58-dev ,…
and for Win we ship the binaries and unpack (ideally not build) them in the repo, the cmake’s file(DOWNLOAD …) is being used already in the repo, see external/*.cmake

…?

heilerm · January 25, 2018, 8:45pm

Correct, wouldn’t we want to manage targets the binaries of all supported targets, albeit in a separate repository? New support could come in the form of a PR. That way, anyone on apple, unix, windows, etc. gets the same repositories, patches and everything. Like Conan’s approach, only nupic would be the user (e.g., ‘boost/1.66@nupic/stable’). CMake can then be conigured to download those binaries into the bin, based on the target triplet, which is pretty much nupic’s, file structure.

I’ll check it out… I thought we might want to just keep those in a separate repository.

chhenning · January 25, 2018, 9:03pm

@breznak S

Sounds good to me. Anything that gets us going should be prioritized. We can also fix later.

breznak · January 25, 2018, 9:03pm

+1 for that being in a separate repo (/just some cloud storage,…)
? not sure if there wouldn’t be conflicts , eg system boost vs ours. But the idea of binary deps for all targets and universal linking to that in Cmake is imho sound.

We had EXACTLY this approach down in the past, with nupic-platform-linux64,... @rhyolight do you recall why did we abandon it? Were there any caveats?

rhyolight · January 25, 2018, 9:49pm

We abandoned those dependency repos because CI environments got more stable and we were able to declare more and more dependencies naturally as the project matured. At some point we stopped needing them.

breznak · January 25, 2018, 9:53pm

@heilerm what if we stick to that too? Unix have typically good package capabilities, for Win we make the binary-deps repo as you suggest. If we see problems with either approach, we can shift to one, or the other solution?

rhyolight · January 25, 2018, 9:59pm

It honestly always felt like the wrong way to do it.

chhenning · January 25, 2018, 10:11pm

@rhyolight What would have been the better way?

rhyolight · January 25, 2018, 10:30pm

When you add binaries to a repo, you’re making assumptions about the environment in which it can be run, therefore you’re also putting possible restrictions on it as well.

Best IMO would be not to have any binaries in any repositories. This may be unrealistic, but it is something to shoot for.

heilerm · January 26, 2018, 12:45am

I think vcpkg will get there. We can even use it to build and export the dependencies we need, dealing with the versioning issue. Once it gets there, zero managed binaries. We will need some sort of manifest, though.

heilerm · February 1, 2018, 10:56pm

I updated my project with a few enhancements:

Each project has its own file structure.
The core header files filter is a bit flatter.
The Python binding project has an example.
The CLR C# project has an example.

I’d like to test this on a Linux box…

The C#, Python, and future projects might need a template in order to hook things up correctly, given any changes to CMake. But… do they even belong in this repository?

Still TODO:

Install
Tests

If you think any of this is wrong, or you want to try something else, let me know.

breznak · February 2, 2018, 1:29pm

A can try to test! Or help you setup Travis for Linux testing.

Topic		Replies	Views
Re-Organisation of the C++, PY Community repos NuPIC Community Fork community , structure , repo , hackers	33	2499	January 25, 2018
Nupic.core Extraction Plan NuPIC planning , nupic-wiki	3	1980	November 30, 2017
So how is progress? NuPIC Community Fork	3	710	April 2, 2019
Cmake update NuPIC question	3	672	October 18, 2018
Nupic now has explicit dependency upon nupic.core NuPIC community	2	771	September 7, 2016

Community Fork - CMake Enhancements and Package Management

Goals

Package Managers

Questions

Related topics