Nupic-core compile error in windows 64bits

Engineering NuPIC
,
1

I tried today to compile the current version of nupic.core under windows with VS2015, CMake3.6.0 and got many errors.
After looking at some of errors I found out the most of problems by compiling all third party SWs, e,g. YamlStaticLib.
For each the third party SW, the following cmake-file is somehows generated automatically in the directory
..\ThirdParty\Stamp\YamlStaticLib\verify-YamlStaticLib.cmake

This file have any rows, that look like:

if("" STREQUAL "")
  message(WARNING "File will not be verified since no URL_HASH specified")
  return()
endif()

and at this code, the compiling has broken, because “the empty string is equal the empty string” too!

After commenting some rows below in this file:

#if("" STREQUAL "")
#  message(WARNING "File will not be verified since no URL_HASH specified")
#  return()
#endif()

#if("" STREQUAL "")
#  message(FATAL_ERROR "EXPECT_VALUE can't be empty")
#endif()

#file("" "C:/Projects/MachineIntelligence/nupic.core-master/external/common/share/yaml/yaml-0.1.5.tar.gz" actual_value)

#if(NOT "${actual_value}" STREQUAL "")
#  message(FATAL_ERROR "error:  hash of
#  C:/Projects/MachineIntelligence/nupic.core-master/external/common/share/yaml/yaml-0.1.5.tar.gz
#does not match expected value
#  expected: ''
#    actual: '${actual_value}'
#")
#endif()

the compiler do somethings until the next error like:

  Performing configure step for 'YamlStaticLib'
  -- The C compiler identification is MSVC 19.0.24210.0
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- broken
  CMake Error at C:/Program Files/CMake/share/cmake-3.6/Modules/CMakeTestCCompiler.cmake:61 (message):
    The C compiler "C:/Program Files (x86)/Microsoft Visual Studio
    14.0/VC/bin/x86_amd64/cl.exe" is not able to compile a simple test program.

Here it trys to test the compiler by compiling a “testCCompiler.c”, but this file dies NOT exist.

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\CL.exe /c /Zi /nologo /W1 /WX- /Od /Ob0 /Oy- /D PSAPI_VERSION=1 /D WIN32 /D _WINDOWS /D _MBCS /D _CRT_SECURE_NO_WARNINGS /D NDEBUG /D _VARIADIC_MAX=10 /D NOMINMAX /D _DEBUG /D “CMAKE_INTDIR="Debug"” /D _MBCS /Gm- /EHs /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR /Fo"cmTC_3c272.dir\Debug\" /Fd"cmTC_3c272.dir\Debug\vc140.pdb" /Gd /TP /analyze- /errorReport:prompt “C:\Projects\MachineIntelligence\nupic.core-master\VS2015\ThirdParty\Build\YamlStaticLib\CMakeFiles\CMakeTmp\testCCompiler.c”

As summary, I think it is a bug in a script that generates a cmake-file for building the third party SW.
I have only basic knowledge about CMake, so that I can not solve this problem properly.

But CMake-Expert can take a time for debugging it well.

Best thanks, Binh

2

Any ideas @breznak, @riccro, or @vkruglikov? (I know these folks have worked on cmake in nupic.core)

3

Hi,

I managed to recreate this problem locally, and found a simple fix that you can try out. In the file CommonCompilerConfig.cmake [1] remove the “/TP” option from the shared_compile_flags variable.

The “/TP” option forces the Microsoft Visual C compiler to assume that all code files will be C++, which causes the issues you’ve seen when building the C-only Yaml and Z libraries.

Removing this option allows CMake to automatically choose the appropriate option (either “/TP” or “/TC”) when compiling source code files.

Regards, Richard.

1 https://github.com/numenta/nupic.core/blob/master/CommonCompilerConfig.cmake#L223

1 Like
4

PS: To get a RelWithDebugInfo nupic_core_solo library built, the following changes are also required;

  • Add #include <iterator> to src/nupic/algorithms/Anomaly.cpp and src/nupic/utils/MovingAverage.cpp (for back_iterator support)
  • Removing “using Serializable::read;” and “using Serializable::write;” from the files src/nupic/math/SparseBinaryMatrix.hpp and src/nupic/math/SparseMatrix.hpp
5

thanks for your comments.
There are some further changes required for buiding successfully as follows:

  1. add Rpcrt4.lib into linker of examples like hello_sp_tp, cpp_region_test etc.
  2. by automatically generating the 3rd party SWs LibYamlCppStatic and LibYamlStatic, the name of the library must be “yaml-cpp.lib” instead of “libyaml-cppmd.lib”

after those changes it works well.

6

Cool! Really good to hear that you have the Visual Studio 2015 version of the nupic.core C++ library built and working again.

I’m unsure why that rpc library is required. I’m guessing it’s required for the serialization code?

I did notice that the Yaml external library building wasn’t/isn’t taking into account of MSVC. And allowing CMake to define whether the ‘lib’ filename prefix is used or not. A patch and/or CMake variable may be required to get CMake to take into account MSVC building, and also whether a library name suffix is required (for the ‘md’, static library linking).

7

hi all,
after updating the current version of nupic.core, I am not able to compile it under windows with VS2015. I had the following errors:
C:\Projects\MachineIntelligence\nupic.core-master\VS2015\ThirdParty\Build\YamlStaticLib\CMakeFiles\CMakeTmp\testCCompiler.c(2):
3>CUSTOMBUILD : fatal error C1189: #error: "The CMAKE_C_COMPILER is set to a C++ compiler"
3> [C:\Projects\MachineIntelligence\nupic.core-master\VS2015\ThirdParty\Build\YamlStaticLib\CMakeFiles\CMakeTmp\cmTC_d6f46.vcxproj]

Maybe some flags for testing compiler are false???

8

I saw

  1. the option “/TP” already exists in “CommonCompilerConfig.cmake”. It must be removed as Richard mentioned some days ago

  2. After removing the option “/TP”, it runs better, but we have LINK-problems as follows:

  • LINK : fatal error LNK1181: cannot open input file ‘C:/Projects/MachineIntelligence/nupic.core-master/VS2015/ThirdParty/Build/YamlStaticLib/yaml.lib’

this library was generated in the last step, but it in the directory ‘release’ like:
‘C:/Projects/MachineIntelligence/nupic.core-master/VS2015/ThirdParty/Build/YamlStaticLib/release/yaml.lib’

  • C:\Projects\MachineIntelligence\nupic.core-master\src\nupic\algorithms\Anomaly.cpp(60): error C3861: ‘back_inserter’: identifier not found
    BitHistory.cpp
    16> Cell.cpp
    16> Unknown compiler version - please run the configure tests and report the results
    16>C:\Projects\MachineIntelligence\nupic.core-master\src\nupic/algorithms/Segment.hpp(319): warning C4838: conversion from ‘double’ to ‘const nupic::Real’ requires a narrowing conversion
    16> Cells4.cpp
    16> Unknown compiler version - please run the configure tests and report the results
    16>C:\Projects\MachineIntelligence\nupic.core-master\src\nupic/algorithms/Segment.hpp(319): warning C4838: conversion from ‘double’ to ‘const nupic::Real’ requires a narrowing conversion
    16> ClassifierResult.cpp
    16> CondProbTable.cpp
    16> Unknown compiler version - please run the configure tests and report the results
    16>C:\Projects\MachineIntelligence\nupic.core-master\src\nupic/math/SparseMatrix.hpp(2747): fatal error C1001: An internal error has occurred in the compiler.
    16> (compiler file ‘f:\dd\vctools\compiler\cxxfe\sl\p1\c\symbols.c’, line 5786)
    16> To work around this problem, try simplifying or changing the program near the locations listed above.
    16> Please choose the Technical Support command on the Visual C++
    16> Help menu, or open the Technical Support help file for more information
    16> C:\Projects\MachineIntelligence\nupic.core-master\src\nupic/math/SparseMatrix.hpp(10459): note: see reference to class template instantiation ‘nupic::SparseMatrix<UI,Real_stor,I,Real_prec,DTZ>’ being compiled
    16> Connections.cpp
    16> Unknown compiler version - please run the configure tests and report the results
    16>C:\Projects\MachineIntelligence\nupic.core-master\src\nupic/math/SparseMatrix.hpp(2747): fatal error C1001: An internal error has occurred in the compiler.

Does anyone have an idea for debugging? Thanks

9

Summary of debugs by building the ThirdParty-SW:

  1. remove “/TP” option
  2. yaml.lib is generated in false directory “/release” so that can NOT be found by linking
  3. In yaml-bundle project, the name “yaml-cpp.lib” for linking is NOT compatible with file name of the library “libyaml-cppmd.lib” generated in the last step

In the files: “Anomaly.cpp” and “MovingAverage.cpp” please insert the neccessary header file: //std::back_inserter
for avoiding compiler error with back_inserter…
This header file existed in the old version, but I really do not know why it is removed by the current version.

I am working on debugging “C1001: An internal error has occurred in the compiler” by compiling SparseMatrix.hpp now, and will come back here if I found out bugs.

10

So this started happening right after the release yesterday? How did you update?

11

Here again, all things we have to do for succesful compiling under windows

  1. remove “/TP” option
  2. yaml.lib is generated in false directory “/release” so that can NOT be found by linking
  3. In yaml-bundle project, the name “yaml-cpp.lib” for linking is NOT compatible with file name of the library “libyaml-cppmd.lib” generated in the last step
  4. In the files: “Anomaly.cpp” and “MovingAverage.cpp” please insert the neccessary header file
  5. remove the compiler option NTA_ASM for projects: nupic_core and nupic_core_solo
  6. Removing “using Serializable::read;” and “using Serializable::write;” from the files src/nupic/math/SparseBinaryMatrix.hpp and src/nupic/math/SparseMatrix.hpp
    as Richard wrote some days ago.

Can anyone please check my proposals again and update the source codes accordingly so that we should not have those problems again?

Best thanks

12

Hey @vkruglikov, it sounds like some of the stuff @thanh-binh.to is talking about above might have been due to the manylinux changes you’ve been doing?

13

I do not update it, I copy it and compile it.

14

These items represent a tidy up of a few issues that have crept in over this year that only affects Microsoft Visual C (MSVC) compilation of the nupic.core library.

Item 1 is restricted to the MSVC path through the CMake scripts [1], and definitely needs removing now that CMake is used to build the external libraries.

Item 2 and 3 is something @vkruglikov could have a look at, and suggest a solution?

Item 4 is due to header include differences between MSVC and GCC toolchains, and the missing include can be added without affecting GCC/Clang.

Item 5 I’m unsure is valid. I’ve not tracked the NTA_ASM use or any recent changes, but feel that it is still required for MSVC. @thanh-binh.to What issues do you face when that is defined?

Item 6 is something that I believe Chetan added that causes the MSVC optimizer to fail, and those two ‘using’ statements can be removed without affecting GCC/Clang.

It looks like all this could be split into a few issues with associated PRs. I can create them this weekend. Further input is required for 2&3 and 5 to determine how those could be solved.

Regards, Richard.

1 https://github.com/numenta/nupic.core/blob/master/CommonCompilerConfig.cmake#L243

1 Like
15

Richard: i will check tomorow Morning item 5, and Report To you

Gesendet mit der 1&1 Mail App

Am 18.08.16 um 18:53 schrieb Richard Crowder

16

@riccro
You are right. I checked it again the item 5 and confirm that no need to remove NTA_ASM. (Sorry)
That mean, for windows NTA_ASM is useless…

17

Thanks for checking that Binh. There are places that use that flag for adding inline assembler, and certainly a flag that needs to be retained and used across operating systems.
I’ll have a look at creating two initial issues; one for the removal of the MSVC /TP flag and optimizer crash (items 1 and 6), plus another to address the missing include files (item 4).
Items 2 and 3 require more care and consideration taking into account other operating systems and build environments.
And double check whether an RPC library is required for certain example applications.

18

I will be catching up with any work you’re doing as soon as I can. I have to work on Bamboo infrastructure probably most of the day today.

19

I am several layers deep into current issues, so I am making a note to follow up soon, but can’t do it right now.

20

I ended up creating two initial PR’s [1, 2] for the invalid compilation flag and missing STL header includes. With one further issue [3] to describe the compiler/optimizer crash we can replicate with the SparseBinaryMatrix and SparseMatrix classes.
One further issue will/may be required to handle the external library building problem. But that will take some time to work out why it occurs with MSVC and not on GCC/CLang builds.

1 https://github.com/numenta/nupic.core/pull/1049
2 https://github.com/numenta/nupic.core/pull/1051
3 https://github.com/numenta/nupic.core/issues/1052

1 Like