Hi @roberrttt, a few thoughts on at least some of your questions in case it’s still useful.
How should I handle the different scales and frequencies of the data?
The different scales involved with different units of measure are handled through the encoding parameters, specifically the resolution param if you’re using the Random Distributed Scalar Encoder (RDSE, which I’d recommend) and min/max values if you use the standard scalar encoder. A good reference is the NAB (Numenta Anomaly Benchmark) code from around 2015, since they applied the same HTM system separately to many different single-feature sensors (including IoT). As I recall they look at a sample of the data for each feature and use the found min/max values plus some padding to get effective ranges, which then help set the resolution.
As far as different frequencies, you can run different HTM models with data coming in at different rates. If you want their outputs to sync up, you’ll need some aggregation or post-processing. The simplest approach—if it’s viable—is to force all measures onto a common frequency during preprocessing. In practice though, many deployments just run a model per stream and synchronize anomaly scores at the alerting layer (e.g., rolling up via Kafka or InfluxDB).
Any other data prep tips?
Industrial sensors can be noisy, so light smoothing or filtering can help without hiding anomalies. HTM doesn’t handle NaNs directly, so you’ll need to impute or flag missing values. And if you have categorical states (like ON/OFF), those can be one-hot or SDR-encoded alongside scalar features.
Is there a specific amount of historical data that I should feed into the system to ensure accurate anomaly detection from the start?
This really depends on how complex your patterns are and how much noise is present. If some features act like simple sine or cosine waves, not much data is needed. As a rule of thumb, for a pattern of length N it helps if the system sees it repeated N times, though this varies a lot with noise and complexity.
One key point is that HTM is continuously learning: if new “normals” emerge later, the model will adapt online without forgetting the old ones. In practice, people usually allow a warm-up phase where alerts are ignored for the first N hours or N thousand points, or they replay historical logs quickly to prime the model before going live.
Integrating with Kafka or MQTT
HTM generally works fine for near-real-time data, but bottlenecks can appear if you try to push very high-frequency, high-dimensional streams into a single model. A common pattern is to run one HTM process per sensor (or per small group of sensors), and let Kafka handle balancing across partitions. That way you scale horizontally and avoid throughput issues.
Once anomalies are detected; how do you recommend setting thresholds?
I’d definitely recommend looking at NAB’s AnomalyLikelihood method. It looks at recent distributions of anomaly scores and spikes when the predictability of the data shifts enough. In industry settings, it’s often combined with operational thresholds like “only alert if anomaly likelihood > 0.999 for 30 seconds” or “if 3+ sensors show anomalies at once.” This helps cut down on false positives.
Final thought
HTM can also give you some explainability: if you log which columns/cells/segments fired at the time of an anomaly, you can sometimes trace it back to the feature or state that caused the spike. That helps engineers trust the alerts.
Best of luck on your project here. I think you’ve found a really powerful anomaly detection method with HTM, and I’d be glad to help if you have follow-up questions.
each with different sampling rates. How should I handle the different scales and frequencies of the data?
