Segmentation Fault while running basic swarm

Engineering NuPIC
,
1

I wanted to just run the basic swarm from the swarm tutorial…
I am getting a segmentation fault. Installed Nupic and Nupic.core properly and ran make successfully.
not sure whats wrong. but this should be pretty straightforward

ubuntu-precision-server:~/numenta/nupic/examples/swarm/simple$ $NUPIC/scripts/run_swarm.py $NUPIC/examples/swarm/simple/search_def.json  --maxWorkers=4
Generating experiment files in directory: /home//numenta/nupic/examples/swarm/simple...
Writing  313 lines...
Writing  114 lines...
done.
None
Successfully submitted new HyperSearch job, jobID=1009
Evaluated 0 models
HyperSearch finished!
Worker completion message: None

Results from all experiments:
----------------------------------------------------------------
Generating experiment files in directory: /tmp/tmpnA5Km4...
Writing  313 lines...
Writing  114 lines...
done.
None
Segmentation fault (core dumped)

The next time I ran it with --overwrite option I see the same core dump

ubuntu-precision-server:~/numenta/nupic/examples/swarm/simple$
 $NUPIC/scripts/run_swarm.py 
$NUPIC/examples/swarm/simple/search_def.json  --maxWorkers=4
Generating experiment files in directory: /home/numenta/nupic/examples/swarm/simple...
Writing  313 lines...
Writing  114 lines...
done.
None
Successfully submitted new HyperSearch job, jobID=1009
Evaluated 0 models
HyperSearch finished!
Worker completion message: None

Results from all experiments:
----------------------------------------------------------------
Generating experiment files in directory: /tmp/tmpnA5Km4...
Writing  313 lines...
Writing  114 lines...
done.
None
Segmentation fault (core dumped)
2 Likes
2

HI Matt – I am following your predicting Sinewave with Nupic tutorial to run a sample swarm. I created a directory sine my home. I ran your generate_data.py code to create the .csv file and also took your search_def.json , then ran the command $NUPIC/scripts/run_swarm.py /home/sine/search_def.json --maxWorkers=4

I do-not understand why it is giving core dump

generate_data.py

import csv
import math

ROWS = 1000

def run():
        fileHandle = open("sine.csv","w")
        writer = csv.writer(fileHandle)
        writer.writerow(["angle","sine"])
        writer.writerow(["float","float"])
        writer.writerow(["",""])

        for i in range(ROWS):
                angle = (i*math.pi)/50.0
                sine_value = math.sin(angle)
                writer.writerow([angle,sine_value])

        fileHandle.close()

if __name__ == "__main__":
        run()

search_def.json

{
 "includedFields": [
    {
      "fieldName": "sine",
      "fieldType": "float",
      "maxValue": 1.0,
      "minValue": -1.0
    }
  ],
  "streamDef": {
    "info": "sine",
    "version": 1,
    "streams": [
      {
        "info": "sine.csv",
        "source": "file://sine.csv",
        "columns": [
          "*"
        ]
      }
    ]
  },
  "inferenceType": "TemporalAnomaly",
  "inferenceArgs": {
    "predictionSteps": [
      1
    ],
    "predictedField": "sine"
  },
  "swarmSize": "medium"
}
1 Like
3

I think we need to identify whether you actually have NuPIC installed properly. Go into your NuPIC checkout folder and run the following commands to run a simple test of NUPIC.

> cd examples/opf/clients/hotgym/simple
> python hotgym.py
INFO:__main__:After 100 records, 1-step altMAPE=38.390240
INFO:__main__:After 200 records, 1-step altMAPE=29.214768
INFO:__main__:After 300 records, 1-step altMAPE=25.829702
INFO:__main__:After 400 records, 1-step altMAPE=24.262638
INFO:__main__:After 500 records, 1-step altMAPE=22.976900
INFO:__main__:After 600 records, 1-step altMAPE=22.890111
INFO:__main__:After 700 records, 1-step altMAPE=22.579927
INFO:__main__:After 800 records, 1-step altMAPE=22.859475
INFO:__main__:After 900 records, 1-step altMAPE=23.299378
INFO:__main__:After 1000 records, 1-step altMAPE=23.396538

You should see output like you see above.

4

There is a segmentation fault with this
ubuntu-precision-server:~/numenta/nupic/examples/opf/clients/hotgym/simple$ python hotgym.py
Segmentation fault (core dumped)…

For compiling and installing Nupic I did this

created a folder numenta where i kept nupic and nupic.core

kaustavsaha@ubuntu-precision-server:~$ mkdir numenta
kaustavsaha@ubuntu-precision-server:~$ cd numenta
kaustavsaha@ubuntu-precision-server:~/numenta$ git clone https://github.com/hstm/nupic.core.git
kaustavsaha@ubuntu-precision-server:~/numenta$ cd nupic.core/
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core$
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core$ git checkout -b gcc4.9-fix
Switched to a new branch 'gcc4.9-fix’
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core$ git pull origin gcc4.9-fix
git clone https://github.com/numenta/nupic.git

I saw all the changes of the 7 files you mentioned

I then deleted build directory and created build and scripts directory

kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core$ mkdir build
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core$ cd build
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core/build$ mkdir scripts
kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core/build$ cd scripts/

kaustavsaha@ubuntu-precision-server:~/numenta/nupic.core/build/scripts$ cmake $NUPIC_CORE -DCMAKE_INSTALL_PREFIX=…/release -DPY_EXTENSIONS_DIR=$NUPIC_CORE/bindings/py/nupic/bindings -DCMAKE_AR:PATH=/usr/bin/gcc-ar -DCMAKE_RANLIB:PATH=/usr/bin/gcc-ranlib

make -j3

make install

Run Tests

cd $NUPIC_CORE/build/release/bin
./cpp_region_test
./unit_tests

Install the Python Bindings

This will expose the C++ nupic.core program to python by installing the nupic.bindings python module.

cd $NUPIC_CORE
python setup.py install --user

NuPIC
Install

cd $NUPIC
python setup.py install --user

Run Tests

First, make sure that py.test is on your path. It got installed into ~/.local/bin in a previous step.

export PATH=~/.local/bin:$PATH

Now run unit tests.

python $NUPIC/scripts/run_nupic_tests.py -u

The unit tests is getting stuck here :
tests/unit/nupic/encoders/random_distributed_scalar_test.py:437: RandomDistributedScalarEncoderTest.testVerbosity PASSED

What am I doing wrong ?

Ubuntu version :15.10
gcc version : 4.9.3

5

Everything before this works fine
Now run unit tests.

python $NUPIC/scripts/run_nupic_tests.py -u

The unit tests is getting stuck here :tests/unit/nupic/encoders/random_distributed_scalar_test.py:437: RandomDistributedScalarEncoderTest.testVerbosity PASSED

6

There is definitely something wrong with your NuPIC installation. I know you are on Ubuntu 15, and you said you were using gcc 4.7. Are you on a VM? At this point, I would suggest that you start fresh using the latest NuPIC code that works with gcc 5.2.

You are ahead of me, so I don’t have a valid wiki document on how to install on Ubuntu 15. You could use this one but ignore the stuff about switching GCC versions. If you get any errors during the installation at all, stop and paste them into pastebin or gist (not in this thread) and link to them here. I’ll try to help.

7

I am on gcc 5.2 now. installation went fine without any errors. cloned nupic and nupic.core

but when i did, i still get segmentation fault

cd examples/opf/clients/hotgym/simple
python hotgym.py

8

@csbond007 I just made some tests on Ubuntu 15.10 with gcc 5.2.1 and Python 2.7.10 and on Ubuntu 16.04 with gcc 5.3.1 and Python 2.7.11+. No segfaults at all.

I’ve created a gist with all required installation steps:

https://gist.github.com/hstm/b375369a22dc63c7a9d469da160f8adc

They are mostly taken from @rhyolight 's instructions (https://github.com/numenta/nupic/wiki/Compiling-NuPIC-on-Ubuntu-14#install-1).

If you follow them carefully step by step, everything should be fine.

9

python $NUPIC/scripts/run_nupic_tests.py -u

The unit tests is getting stuck here :tests/unit/nupic/encoders/random_distributed_scalar_test.py:437: RandomDistributedScalarEncoderTest.testVerbosity PASSED

10

OK. Please execute this in your terminal:

gdb -ex r --args python $NUPIC/scripts/run_nupic_tests.py -u

If the process hangs or segfaults, we should get some information.

Please follow @rhyolight 's advice and paste the output into pastebin or gist (not in this thread).

11

I had success on Ubuntu server 16.04 and mysql 5.7.12 with the swarming tests, using pull request https://github.com/numenta/nupic.core/pull/984.

  1. Run the swarming tests

     $ ./scripts/run_nupic_tests.py -w

  2. run hotgym

~/nta/nupic/examples/opf/clients/hotgym/simple$ python hotgym.py
INFO:__main__:After 100 records, 1-step altMAPE=23.183145
INFO:__main__:After 200 records, 1-step altMAPE=21.548877
INFO:__main__:After 300 records, 1-step altMAPE=21.227594
INFO:__main__:After 400 records, 1-step altMAPE=20.686270
INFO:__main__:After 500 records, 1-step altMAPE=20.417234
INFO:__main__:After 600 records, 1-step altMAPE=20.852339
INFO:__main__:After 700 records, 1-step altMAPE=20.907660
INFO:__main__:After 800 records, 1-step altMAPE=21.137106
INFO:__main__:After 900 records, 1-step altMAPE=20.875763
INFO:__main__:After 1000 records, 1-step altMAPE=20.789707
~/nta/nupic/examples/opf/clients/hotgym/simple$ echo $?
0

@rhyolight, I moved the conversation about mysql user configuration and NTA_CONF_PATH to its own thread: When swarming, got ERROR 1698 (28000): Access denied for user 'root'@'localhost' mysql

12

I am running on a AWS free tier Ubuntu box. OS version is Ubuntu 14.04.4 LTS
Default python version was 2.7.6.

Failed building wheel for pycapnp

sudo apt-get update -y
sudo apt-get install git g++ cmake python-dev -y
git clone https://github.com/numenta/nupic.core.git
git clone https://github.com/numenta/nupic.git
export NUPIC=$HOME/nupic

13

I don’t think this is necessary, because the current location of the nupic-default.xml file is the default location NuPIC will look for it. I remember updating this a couple years ago. Hopefully this is still the case?

I found MySQL mentioned on the Running Swarms page, and there is also a MySQL Settings page. Maybe we need something more comprehensive?

14

Tried with 2.7.11 also
sudo add-apt-repository ppa:fkrull/deadsnakes-python2.7
sudo apt-get update
sudo apt-get upgrade

ubuntu@ip-172-31-12-69:~$ python --version
Python 2.7.11

on running curl https://bootstrap.pypa.io/get-pip.py | sudo python
i am getting

The directory ‘/home/ubuntu/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.

Same errors as before

15

@csbond007 From your logs:

virtual memory exhausted: Cannot allocate memory

You need more RAM!

16

AWS Ubuntu 14 – @rhyolight – wil it work with 2 GB RAM ?

Ubuntu 15 – @helge – gdb -ex r --args python $NUPIC/scripts/run_nupic_tests.py -u
even with this it hangs at

tests/unit/nupic/encoders/random_distributed_scalar_test.py:437: RandomDistributedScalarEncoderTest.testVerbosity PASSED

17

@helge

kaustavsaha@ubuntu-precision-server:~/nupic/examples/opf/clients/hotgym/simple$ gdb -ex r --args python hotgym.py -u
GNU gdb (Ubuntu 7.10-1ubuntu2) 7.10

This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from python…(no debugging symbols found)…done.
Starting program: /usr/bin/python hotgym.py -u
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7ffff25cb700 (LWP 25223)]
[New Thread 0x7ffff1dca700 (LWP 25224)]
[New Thread 0x7fffed5c9700 (LWP 25225)]

Program received signal SIGSEGV, Segmentation fault.
__memset_sse2 () at …/sysdeps/x86_64/multiarch/…/memset.S:78
78 …/sysdeps/x86_64/multiarch/…/memset.S: No such file or directory.
(gdb)

18

@helge

The backtrace

19

I don’t know. I would try with 4GB. The segfault is very likely because of this installation problem, so focus on that first.

20

Select at least an m3.medium instance (Micro instance fails to compile due to insufficient memory)

An m3.medium has 3.75 GB RAM

1 Like