@Bitking I want to understand the details of your logic for dynamically update the anomaly likelihood historicWindowSize. I remember you saying that the historicWindowSize is updated under certain conditions, can I ask you to lay out that logic for me once more? Thank you very much 
1 Like
The sample periods are the blue/white dotted line, with a sub-sample being taken on each transition from blue to white or white to blue.Each sub-sample is combined with eight other samples and the sum divided by eight. This provides some reduction in random variations.We call the bigger sample period a boxcar.
The signal we are sampling has a step from 0 to 10 somewhere during sub-sample eleven.
This poisons the second boxcar as it shows an output that is not 0 or 10 but some intermediate value that is neither correct value.
What we do for the second line is keep a running tally of the best guess of the true value (the prior value) and some programmable delta value. If the current sub-sample is more than this delta from the last boxcar we throw this sub-sample away and terminate this boxcar. We display the last known value (the prior boxcar) and start a new boxcar. After 8 samples we update the display with this new value. This gets us to display a new correct value faster without the display of the funky intermediate value.
As you can imagine we can do this same logic with a rolling average where the boxcar is the sum of the last n readings divided by N. There are tweaks on this theme for computational efficiency with longer buffers but they are not germane for this exposition.
A slightly different but related solution is that we keep track of the number of readings we have accumulated in the current boxcar divided by the number of sample that have been accumulated until we discover a delta value. We still throw out the “bad” sub-sample. The display is updated with the running computation between changes; with this method the display gets more stable as the sample period gets longer.
While this does not directly address the problem you are describing in the hangout the takeaway is that you can fiddle with your sample periods and resets to make sure that your sample is adjusted to the period and learning state that produces the desired behavior.
For example - you can run a delayed stream and use the output from a real-time “monitor stream” to trigger a learning reset if you want to respond to some fast changing transient signal.
The method: Assume two detectors, one tuned for a particular very sensitive setting (small anomaly value) and one with a very large setting.
You insert a "delay line buffer before the sensitive detector. That is - a circular buffer of whatever period that you are concerned about - say 4000 samples. You monitor the stream in real-time with the coarse detector. If it does something interesting you may decide to turnoff learning on the delayed sensitive stream so you don’t learn a large error. In essence - you know what is going to happen before it does.
Or you could time the learning to start at some predetermined part of a wave-form if your events are happening at irregular intervals and you want to learn and compare event to event.
There is a lot you can do with this type of thinking.
3 Likes