Segmentation Fault while running basic swarm

[vkruglikov]

The real root cause, as I discovered during my work on the manylinux nupic.bindings wheel, was due to the unanticipated confluence of runtime symbol preemption and c++ ABI incompatibility between nupic.bindings and pycapnp. I am going to document it here for “posterity”:

1. When you encountered the SEGFAULT, you were running on Ubuntu 16.04, whose system headers and libraries were created using the updated c++11 ABI.
2. capnproto sources in pycapnp are compiled upon installation using the Ubuntu 16.04 toolchain with those new c++11 ABI
3. nupic.bindings was obtained either from from Numenta’s S3 or built elsewhere on either Ubuntu 12.04 or 14.04, using an older c++ ABI that is incompatible with the one on Ubuntu 16.04 (where pycapnp was built).
4. nupic.bindings includes its own copy of capnproto c++ sources that is very close to the one included in pycapnp’s python extension. However, nupic.bindings’s capnproto c++ code was compiled as part of nupic.bindings using the older toolchain.
5. Neither pycpanp nor nupic.bindigns were hiding their symbols, so all the symbols in pycapnp and nupic.bindings extensions were public, including the similar capnproto symbols, subjecting both libraries to symbol preemption during runtime linking.
6. Notice in the stack trace quoted below that control from the destructor capnp::SchemaLoader::Impl::~Impl in pycapnp extension (capnp.so) is inadvertently transferred to methods compiled into the nupic.bindings extension _math.so. Recall that pycapnp’s capnp.so and nupic.bindings’ _math.so were compiled on different platforms using incompatible c++ ABI. This explains the SEGFAULT on Ubuntu 16.04 (pycanp and nupic.bindings extensions were compiled using INCOMPATIBLE c++ ABI) and no SEGFAULT on Ubuntu 14.04 (pycanp and nupic.bindings extensions were compiled using compatible c++ ABI)

Simple as that!

@csbond007

(gdb) bt
#0  __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:78
#1  0x00007fffe938980c in std::_Hashtable<unsigned long, std::pair<unsigned long const, capnp::SchemaLoader::Impl::RequiredSize>, std::allocator<std::pair<unsigned long const, capnp::SchemaLoader::Impl::RequiredSize> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/nupic/bindings/_math.so
#2  0x00007fffe9385dbe in std::_Hashtable<unsigned long, std::pair<unsigned long const, capnp::SchemaLoader::Impl::RequiredSize>, std::allocator<std::pair<unsigned long const, capnp::SchemaLoader::Impl::RequiredSize> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::~_Hashtable() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/nupic/bindings/_math.so
#3  0x00007fffe937e81c in std::unordered_map<unsigned long, capnp::SchemaLoader::Impl::RequiredSize, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, capnp::SchemaLoader::Impl::RequiredSize> > >::~unordered_map() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/nupic/bindings/_math.so
#4  0x00007fffe9392b24 in capnp::SchemaLoader::Impl::~Impl() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/nupic/bindings/_math.so
#5  0x00007fffe9392b96 in kj::_::HeapDisposer<capnp::SchemaLoader::Impl>::disposeImpl(void*) const () from /home/kaustavsaha/.local/lib/python2.7/site-packages/nupic/bindings/_math.so
#6  0x00007fffe678c973 in capnp::SchemaLoader::~SchemaLoader() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/capnp/lib/capnp.so
#7  0x00007fffe670b6ae in capnp::compiler::Compiler::Impl::clearWorkspace() () from /home/kaustavsaha/.local/lib/python2.7/site-packages/capnp/lib/capnp.so
#8  0x00007fffe670b7f0 in capnp::compiler::Compiler::clearWorkspace() const () from /home/kaustavsaha/.local/lib/python2.7/site-packages/capnp/lib/capnp.so