Error metrics on swarming

Engineering NuPIC
1

I saw this Wiki! https://github.com/numenta/nupic/wiki/Running-Swarms
I want to use another error metrics (Root Mean Squared Error, etc) on swarming at hot gym demo.
Which and Where code should I update?

1 Like
2

@scott @subutai Do you know if this is possible?

3

I’m not sure how easy it is to use your own error metric. There is a notion of different metrics and there are at least implementations of average absolute error (aae) and mean average percent error (altMAPE). You can see them specified in a MetricSpec in the description.py here:
https://github.com/numenta/nupic/blob/386ef719ba54861f70360ebb287abd845f2fed03/examples/opf/experiments/multistep/hotgym/description.py#L399-L414

I believe the swarm will use the first metric for swarming purposes but it looks like you may also be able to specify the swarm metric in the permutations.py:
https://github.com/numenta/nupic/blob/386ef719ba54861f70360ebb287abd845f2fed03/examples/opf/experiments/multistep/hotgym/permutations.py#L77

But it’s been a while since I last ran swarms so it would take some digging and experimentation to figure out if you can swap in your own metric.

4

It’s also been a while since I looked into this but swarming is designed to be able to use your own metrics. If you look at nupic/frameworks/opf/metrics.py you will see there is an interface for metrics and a bunch of already defined existing metrics, including RMSE.

I don’t know how easy it is - you’ll have to figure out how to adapt permutations.py per Scott’s email to specify the metric. This would be a nice example for someone to do and augment the basic swarming example.