Problem Running HTMEngine Skeleton App

[htmhs]

Hi,

I’m trying out the HTM engine Skeleton App. Everything seems to work as expected, except the fact that I’m not getting the raw_anomaly_score and anomaly_score fields populated in the mysql database.

However, querying the metric table indicates that everything should be working:

select uid, name, description, status from metric where name = ‘cpu_percent’;
±---------------------------------±------------±--------------------------±-------+
| uid | name | description | status |
±---------------------------------±------------±--------------------------±-------+
| 830b1d61c724452cac2bdcd2e7371570 | cpu_percent | Custom metric cpu_percent | 1 |
±---------------------------------±------------±--------------------------±-------+

The anomaly service log indicates the following:

ID=830b1d61c724452cac2bdcd2e7371570, METRIC=cpu_percent, STATUS=1, SERVER=cpu_percent>
2016-09-15 17:58:59,595 - htmengine.anomaly(74418) - WARNING - VER=0.0.0 Received command result=ModelCommandResult<commandID=5e276f69b1a944a6b1077a1ef3ceff1a, method=defineModel, status=0> for model=830b1d61c724452cac2bdcd2e7371570 of metric=<METRICID=830b1d61c724452cac2bdcd2e7371570, METRIC=cpu_percent, STATUS=1, SERVER=cpu_percent> that was not in CREATE_PENDING state
2016-09-15 17:59:00,725 - htmengine.anomaly(74418) - INFO - VER=0.0.0 Model was created for <METRICID=830b1d61c724452cac2bdcd2e7371570, METRIC=cpu_percent, STATUS=1, SERVER=cpu_percent>
2016-09-15 17:59:00,725 - htmengine.anomaly(74418) - WARNING - VER=0.0.0 Received command

I’m running on OS X.

Any thoughts on what might be causing the issue?

[vkruglikov]

Can you share more of the log, including both immediately before and after?

[vkruglikov]

@htmhs

that was not in CREATE_PENDING state

The reason for the warning is that either you tried promote metric 830b1d61c724452cac2bdcd2e7371570 multiple times, or something crashed after anomaly service changed its status to 1 (ACTIVE) and the original ModelCommandResult message didn’t get ACK’d, so RabbitMQ redelivered it. You might want to look further back in the log for 830b1d61c724452cac2bdcd2e7371570 to get a better picture.

There are several services involved in making htmegine work: metric_listener, metric_storer, model_scheduler, and anomaly_service. If you’re sending data rows to htmengine via TCP/IP socket, then it would be going to metric_listener then to metric_storer, then to model_scheduler for HTM processing, and then to anomaly_service for final anomaly post-processing.

You need to follow that data path by looking at each service’s logs in turn to see where the problem might be happening. Also, is the RabbitMQ service up and running? RabbitMQ has its own logs that you could examine for issues.

[vkruglikov]

The supervisor log (in same directory as the other htmengine service logs) is a good central place to look to see if services may be misbehaving (i.e., crashing).

Also, the supervisorctl status command is handy for examining current status of services (possibly with additional options, such as port number, depending on your your configuration)

[htmhs]

Thanks for your response.
I ran everything from scratch. All services run (i.e no crashes) according to supervisor status. However, model_scheduler.log indicates the following error: [I’ve manually replaced my actual path with <MY_PATH> in the following text]

2016-09-17 00:19:04,211 - htmengine.model_runner(10322) - ERROR - VER=0.0.0  {TAG:SWAP.MR.STOP.ABORT}
Traceback (most recent call last):
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'
2016-09-17 00:19:04,315 - htmengine.model_swapper.slot_agent(10304) - ERROR - VER=0.0.0  SlotAgent<slotID=2, modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.SA.MODEL.STOP.DONE} modelState=_CurrentModelState<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa, stopPend=True, stopReq=False, modelFailed=False, exitStatus=1, modelRunner=ModelRunnerProxy<model=15b221c6a3ed4dfb9ba2a73bb2d983fa, pid=10322, returnCode=1>>
2016-09-17 00:19:04,332 - htmengine.model_swapper.slot_agent(10304) - INFO - VER=0.0.0  SlotAgent<slotID=2, modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.SA.MODEL.STARTED} modelState=_CurrentModelState<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa, stopPend=False, stopReq=False, modelFailed=False, exitStatus=None, modelRunner=ModelRunnerProxy<model=15b221c6a3ed4dfb9ba2a73bb2d983fa, pid=10337, returnCode=None>>
2016-09-17 00:19:04,870 - htmengine.model_runner(10337) - INFO - <VER=0.0.0, SERVICE=MRUN>   {TAG:SWAP.MR.START} argv=['<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py', '--modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa']
2016-09-17 00:19:04,900 - htmengine.model_runner(10337) - INFO - VER=0.0.0  ModelRunner<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: Processing model command: ModelCommand<commandID=34b0bf15e59c4088b0ae38e435039b17, method=defineModel>
2016-09-17 00:19:04,901 - htmengine.model_runner(10337) - WARNING - VER=0.0.0  defineModel: modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa already exists with same creation meta data (side-effect of at-least-once delivery guarantee?)
2016-09-17 00:19:05,938 - htmengine.model_runner(10337) - INFO - VER=0.0.0  ModelRunner<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.MR.FINAL.SUMMARY} totalBatches=2; totalRequests=8; totalDupBatches=0; duration=1.06675505638
2016-09-17 00:19:05,939 - htmengine.model_runner(10337) - ERROR - VER=0.0.0  {TAG:SWAP.MR.STOP.ABORT}
Traceback (most recent call last):
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'
2016-09-17 00:19:06,043 - htmengine.model_swapper.slot_agent(10304) - ERROR - VER=0.0.0  SlotAgent<slotID=2, modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.SA.MODEL.STOP.DONE} modelState=_CurrentModelState<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa, stopPend=True, stopReq=False, modelFailed=False, exitStatus=1, modelRunner=ModelRunnerProxy<model=15b221c6a3ed4dfb9ba2a73bb2d983fa, pid=10337, returnCode=1>>
2016-09-17 00:19:06,050 - htmengine.model_swapper.slot_agent(10304) - INFO - VER=0.0.0  SlotAgent<slotID=2, modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.SA.MODEL.STARTED} modelState=_CurrentModelState<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa, stopPend=False, stopReq=False, modelFailed=False, exitStatus=None, modelRunner=ModelRunnerProxy<model=15b221c6a3ed4dfb9ba2a73bb2d983fa, pid=10348, returnCode=None>>
2016-09-17 00:19:06,526 - htmengine.model_runner(10348) - INFO - <VER=0.0.0, SERVICE=MRUN>   {TAG:SWAP.MR.START} argv=['<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py', '--modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa']
2016-09-17 00:19:06,537 - htmengine.model_runner(10348) - INFO - VER=0.0.0  ModelRunner<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: Processing model command: ModelCommand<commandID=34b0bf15e59c4088b0ae38e435039b17, method=defineModel>
2016-09-17 00:19:06,537 - htmengine.model_runner(10348) - WARNING - VER=0.0.0  defineModel: modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa already exists with same creation meta data (side-effect of at-least-once delivery guarantee?)
2016-09-17 00:19:07,615 - htmengine.model_runner(10348) - INFO - VER=0.0.0  ModelRunner<modelID=15b221c6a3ed4dfb9ba2a73bb2d983fa>: {TAG:SWAP.MR.FINAL.SUMMARY} totalBatches=2; totalRequests=8; totalDupBatches=0; duration=1.08736610413
2016-09-17 00:19:07,616 - htmengine.model_runner(10348) - ERROR - VER=0.0.0  {TAG:SWAP.MR.STOP.ABORT}
Traceback (most recent call last):
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 886, in <module>
    main(sys.argv)
  File "<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py", line 872, in main
    runner.run()
  File "/Library/Python/2.7/site-packages/nupic/support/decorators.py", line 58, in exceptionLoggingWrap
    logger.exception(
AttributeError: 'function' object has no attribute 'exception'

metric_listener does not indicate any errors; I see the following info message:

2016-09-17 00:29:04,895 - __main__(11645) - INFO - VER=0.0.0  Starting with host=0.0.0.0, port=2003, protocol=plain, transport=tcp

metric_storer, also, does not show any errors. I see the following line(s) when send_cpu.py is running:

2016-09-17 00:29:48,010 - __main__(11647) - INFO - VER=0.0.0  Processing 1 records for 1 models from 1 batches.

The "metric_value" column in "metric_data table" is populated with values, but anomaly numbers don't show up (after create_cpu_percent_model.py is run once). Here's a snapshot of the table:

+----------------------------------+-------+---------------------+--------------+-------------------+---------------+---------------+
| uid                              | rowid | timestamp           | metric_value | raw_anomaly_score | anomaly_score | display_value |
+----------------------------------+-------+---------------------+--------------+-------------------+---------------+---------------+
| 15b221c6a3ed4dfb9ba2a73bb2d983fa |     1 | 2016-09-16 23:48:06 |          5.4 |              NULL |          NULL |          NULL |
| 15b221c6a3ed4dfb9ba2a73bb2d983fa |     2 | 2016-09-16 23:48:11 |          5.8 |              NULL |          NULL |          NULL |
| 15b221c6a3ed4dfb9ba2a73bb2d983fa |     3 | 2016-09-16 23:48:16 |          8.4 |              NULL |          NULL |          NULL |
| 15b221c6a3ed4dfb9ba2a73bb2d983fa |     4 | 2016-09-16 23:48:21 |            5 |              NULL |          NULL |          NULL |
| 15b221c6a3ed4dfb9ba2a73bb2d983fa |     5 | 2016-09-16 23:48:26 |          7.1 |              NULL |          NULL |          NULL |
| 15

RabbitMQ is also running. I get the following when I list queue names:

+----------------------------------------------------------------+
|                              name                              |
+----------------------------------------------------------------+
| skeleton.metric.custom.data                                    |
| skeleton.mswapper.model.input.15b221c6a3ed4dfb9ba2a73bb2d983fa |
| skeleton.mswapper.results                                      |
| skeleton.mswapper.scheduler.notification                       |
+----------------------------------------------------------------+

I appreciate your help.

[vkruglikov]

@htmhs

logger.exception(
AttributeError: ‘function’ object has no attribute ‘exception’

cc @rhyolight

@htmhs, there is clearly a failure, but it’s getting masqueraded by an erroneous use of the logExceptions decorator in several scripts inside htmengine. I just submitted the pull request https://github.com/numenta/numenta-apps/pull/894 to fix the issue with logExceptions. It’s waiting for tests to complete and for a reviewer to approve my PR.

In the meantime, if you would like to try running with my PR (and if it works), you might be able to learn the actual cause of the failure as reported by the properly-configured logExceptions decorator.

Best,
Vitaly

[vkruglikov]

@htmhs, https://github.com/numenta/numenta-apps/pull/894 has now been merged, so you can try it directly from the master branch to help discover what is really failing.

[htmhs]

Great, Thanks!
I’ll try it out and will let you know shortly.

[htmhs]

Ok, with the new update, I’m not getting raw data in the DB anymore. I see the following error in metric_storer.log:

OperationalError: (OperationalError) (1054, “Unknown column ‘metric_data.multi_step_best_predictions’ in ‘field list’”) ‘SELECT metric_data.uid, metric_data.rowid, metric_data.timestamp, metric_data.metric_value, metric_data.raw_anomaly_score, metric_data.anomaly_score, metric_data.display_value, metric_data.multi_step_best_predictions \nFROM metric_data \nWHERE metric_data.uid = %s AND metric_data.rowid = %s ORDER BY metric_data.rowid ASC’ (‘73eeda4ab37640ceb89bd35b91111f6f’, 0L)

This is clearly due to the fact that metric_data.multi_step_best_predictions is not part of the DB schema. I should mention that I had manually created the schema using the instructions from here : https://github.com/numenta/numenta-apps/tree/master/htmengine (under the repository section). The reason that I tried to manually create the schema was a failure of the migration script with the following error:

sqlalchemy.exc.OperationalError: (OperationalError) (1050, “Table ‘instance_status_history’ already exists”) “\nCREATE TABLE instance_status_history (\n\tserver VARCHAR(100) DEFAULT ‘’ NOT NULL, \n\ttimestamp TIMESTAMP DEFAULT ‘0000-00-00 00:00:00’, \n\tstatus VARCHAR(32) DEFAULT ‘’ NOT NULL, \n\tPRIMARY KEY (server, timestamp)\n)\n\n” ()

I tried changing the default value for timestamp field (which seems to be introducing the problem) in the script, but that didn’t help and I decided to create the schema manually using the mentioned instructions. That worked, as far as getting raw data is concerned.

Interestingly, I used to receive data values before, but that isn’t the case anymore after pulling the latest changes from the numenta-apps repository. Did anything else change in addition to the issue with the logger?

Any suggestions on fixing the migration issue or updates on how I should create a new schema with metric_data.multi_step_best_predictions column?

[htmhs]

Sorry for the confusion, but I was not receiving the following error:

sqlalchemy.exc.OperationalError: (OperationalError) (1050, “Table ‘instance_status_history’ already exists”) “\nCREATE TABLE instance_status_history (\n\tserver VARCHAR(100) DEFAULT ‘’ NOT NULL, \n\ttimestamp TIMESTAMP DEFAULT ‘0000-00-00 00:00:00’, \n\tstatus VARCHAR(32) DEFAULT ‘’ NOT NULL, \n\tPRIMARY KEY (server, timestamp)\n)\n\n” ()

I received a different one related to incorrect default value for the timestamp field. I’ll start from scratch and will re-run migration, while waiting for your response.

[htmhs]

Ok, so applying migration on a new install gives the following error:

sqlalchemy.exc.OperationalError: (OperationalError) (1067, “Invalid default value for ‘timestamp’”) “\nCREATE TABLE instance_status_history (\n\tserver VARCHAR(100) DEFAULT ‘’ NOT NULL, \n\ttimestamp TIMESTAMP DEFAULT ‘0000-00-00 00:00:00’, \n\tstatus VARCHAR(32) DEFAULT ‘’ NOT NULL, \n\tPRIMARY KEY (server, timestamp)\n)\n\n” ()

[vkruglikov]

Which version of mysql are you running?

[htmhs]

This is the output of “mysql --version”

mysql Ver 14.14 Distrib 5.7.15, for osx10.11 (x86_64) using EditLine wrapper

[vkruglikov]

cc @rhyolight

@htmhs, I think I see what’s going on. Recent versions of mysql (5.7.8+ ?) have defaulted to more strictness, disallowing certain defaults. mysql has been going back and forth on this for some time.

https://github.com/numenta/numenta-apps/pull/781 fixed up the migrations for the apps in github.com/numenta/numenta-apps to be compatible. Look at the three files with skeleton-app/migrations/versions/ in their path in https://github.com/numenta/numenta-apps/pull/781/files.

I am guessing that you’re using the skeleton app from https://github.com/htm-community/skeleton-htmengine-app. This community project has not been updated to be compatible with the defaults in the recent versions of mysql, hence the mysql errors that you’re seeing.

It would be great to fix the migrations in https://github.com/htm-community/skeleton-htmengine-app.

Short of fixing it, you can try to work around this by overriding the default mysql modes when starting mysql. The problem is with NO_ZERO_DATE. I ran this query in mysql shell to get all the default modes: SELECT @@sql_mode;. Here is example output: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Then, I removed NO_ZERO_DATE and passed the remaining settings as the value of --sql-mode option to mysql.server start.

For example, I had this function in my ~/.bashrc:

function mysqlstart()
{
 # exclude NO_ZERO_DATE, because it breaks nupic swarning tests and htmengine.
 mysql_modes="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
 mysql.server start --sql-mode=${mysql_modes}
}

Hope this helps,
Vitaly

[htmhs]

Thanks for your response. I tried bypassing the strictness. My sql_mode configuration looks like below:

mysql> select @@GLOBAL.sql_mode;
+------------------------------------------------------------------------------------------------------------------------------+
| @@GLOBAL.sql_mode                                                                                                            |
+------------------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select @@SESSION.sql_mode;
+------------------------------------------------------------------------------------------------------------------------------+
| @@SESSION.sql_mode                                                                                                           |
+------------------------------------------------------------------------------------------------------------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |
+------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

That’s all set, and migration seems to happen without errors. Here are the tables that get created, and the description of metric_data table:

mysql> show tables;
+-------------------------+
| Tables_in_skeleton      |
+-------------------------+
| alembic_version         |
| instance_status_history |
| lock                    |
| metric                  |
| metric_data             |
+-------------------------+
5 rows in set (0.00 sec)

mysql> describe metric_data;
+-------------------+-------------+------+-----+---------+-------+
| Field             | Type        | Null | Key | Default | Extra |
+-------------------+-------------+------+-----+---------+-------+
| uid               | varchar(40) | NO   | PRI |         |       |
| rowid             | int(11)     | NO   | PRI | 0       |       |
| timestamp         | datetime    | NO   | MUL | NULL    |       |
| metric_value      | double      | NO   |     | NULL    |       |
| raw_anomaly_score | double      | YES  |     | NULL    |       |
| anomaly_score     | double      | YES  | MUL | NULL    |       |
| display_value     | int(11)     | YES  |     | NULL    |       |
+-------------------+-------------+------+-----+---------+-------+
7 rows in set (0.00 sec)

But there’s still an issue with the schema. I get the following error in metric_storer.log when send_cpu.py runs:

OperationalError: (OperationalError) (1054, "Unknown column 'metric_data.multi_step_best_predictions' in 'field list'") 'SELECT metric_data.uid, metric_data.rowid, metric_data.timestamp, metric_data.metric_value, metric_data.raw_anomaly_score, metric_data.anomaly_score, metric_data.display_value, metric_data.multi_step_best_predictions \nFROM metric_data \nWHERE metric_data.uid = %s AND metric_data.rowid = %s ORDER BY metric_data.rowid ASC' ('0f8c3350d68746b6b8386bd21b2ea9f2', 0L)

Is there a different migration script that I should use?

[vkruglikov]

@htmhs, I thought https://github.com/numenta/numenta-apps/pull/894 was merged earlier, but it wasn’t. I just merged it. Sorry for the confusion.

[htmhs]

Thanks for your response.

I updated my branch of numenta_apps, and ran the migration script in my branch of skeleton-htmengine-app again. It runs successfully but still, the metric_data table does not include multi_step_best_predictions column.

Is there a different skeleton-app that I should use? If not, could you please point me to the right database schema or the right migration script?

[vkruglikov]

@htmhs, I created this skeleton-htmengine-app issue in github that describes the problem and how to create a pull request that fixes the problem with metric_data.multi_step_best_predictions and Invalid default value for 'timestamp'.

[htmhs]

Thanks for your response. I went through the steps and am now getting metric_value data in the database, which seems to have the right schema. I’m not getting anomaly scores though.

I see the following error in model_scheduler.log:

2016-09-22 18:22:34,509 - htmengine.model_runner(9620) - ERROR - VER=0.0.0 Unhandled exception TypeError(‘1.0 is not JSON serializable’,) from <function run at 0x109a747d0>. Caller stack:
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py”, line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py”, line 892, in
main(sys.argv)
File “<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py”, line 878, in main
runner.run()
File “<MY_PATH>/numenta-apps/nta.utils/nta/utils/error_handling.py”, line 66, in exceptionLoggingWrap
sys.exc_info()[1], func, ‘’.join(traceback.format_stack()), )
Traceback (most recent call last):
File “<MY_PATH>/numenta-apps/nta.utils/nta/utils/error_handling.py”, line 62, in exceptionLoggingWrap
return func(*args, **kwargs)
File “<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_runner.py”, line 244, in run
results=results)
File “<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_swapper_interface.py”, line 973, in submitResults
batchState=BatchPackager.marshal(batch=results))
File “<MY_PATH>/numenta-apps/htmengine/htmengine/model_swapper/model_swapper_interface.py”, line 496, in marshal
return json.dumps([o.getstate() for o in batch])
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”, line 243, in dumps
return _default_encoder.encode(obj)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py”, line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py”, line 270, in iterencode
return _iterencode(o, 0)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py”, line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: 1.0 is not JSON serializable

I appreciate any thoughts or suggestions you might have to address the issue.

Thanks for your help.

[vkruglikov]

@htmhs, could you please provide a tiny python script that sends the minimal data to skeleton app that reproduces that traceback?

Thank you.