Using a Classifier with htm.core for Skeleton-Based Human Activity Recognition

Engineering NuPIC
1

Hello Everyone,

I am working on a research project involving Human Activity Recognition using the NTU RGB+D 3D skeleton dataset, and I am currently building a pipeline with htm.core to process the skeletal data. So far, I have implemented encoders for 3D joint coordinates and have successfully passed the encoded SDRs through Spatial Pooler and Temporal Memory components of htm.core.

Now, I am looking to implement/integrate a classifier preferably something like the SDRClassifier or any HTM-compatible approach—to map the temporal memory outputs to activity labels (e.g., walking, sitting, falling, etc.). However, I couldn’t find a clear example or guide within the htm.core repository on how to do this.

  • Is the SDRClassifier available or supported in the current version of htm.core?
  • If not, what are some recommended ways to implement a classifier that works well with the SDR outputs from Temporal Memory?

Any help or direction would be greatly appreciated. Thanks

1 Like
2

A meaningful result could be possible by training any classifier on the activation state of each cell.
If TM is organized as (e.g.) 1000 columns by 16 cells each, then you can compound a SDR of 16000 bits which can be used as training input to the classifier.

Which classifier algorythm?
Limits might be imposed by:

  • how long training time/how big a hardware can you afford - because the input space bit vector could be quite large
  • whether you need continuous online training or not.

I think the SDRClassifier in particular is a simple linear classifier which is designed to be fast on SDR inputs and is online too, it can be fed one SDR at a time for both inference and training.
Its limits are simplicity of linear classifying which is equivalent with a fully connected NN with no hidden layers. So you might be tempted to test any other classifier from (e.g.) ScikitLearn toolkit like random forest, NN, etc.

3

Welcome @muhammadzulqarnain58! One idea is to train separate models for each activity type (walking, sitting, falling, etc). Then feed the incoming behaviors to all models and continuously compare their anomaly scores. If one of the models dips in anomaly scores relative to the others it means that model is suddenly less surprised/more familiar with what it’s seeing – suggesting that the activity matches what it was trained on. So if the walking model dips in anomaly scores maybe the incoming activity is walking, but if the sitting model then dips down lower maybe the activity has shifted to sitting. If the anomaly scores are too volatile maybe a rolling average to smoothen it out. By tracking the anomaly scores for each activity-type model you’d have a way to estimate how likely each activity type was at each time, and what points in time the activity seems to have shifted.