About integration of Flink-HTM

Hello!
Honestly I didn’t know where post these questions so I put here because I’m using java. Please move it if there is a better category.
I prefer to post a pastebin pic in order to explain my problem because I wrote a bit of code lines and it looks better on it.
Briefly: I have a Kafka broker and cluster with running Flink instance. I want to perform anomaly detection with HTM on the cluster feeding it with data coming from Kafka.

I have downloaded Flink-HTM from github and explored it to understand.

Problem

P.S.: If can help, I can attach also my custom deserializer and custom timestamp extractor code.

Thanks in advance,
Andrea

Calling @EronWright

Yes I already contacted him via e-mail. I hoped there was other experts here :slight_smile:

1 Like

Hi @Andrea_Giordano,

The “second” value is the most likely value previously submitted to the HTM which is stored by the HTM, and correlates to the next predicted value (not input encoding). The HTM emits a prediction binary vector (SDR), and that vector is the HTM’s encoding for the Actual Value submitted to the HTM. The two are stored together so that actual (human readable) values may be retrieved as prediction results.

The XXXClassifier, is an algorithm added to the biological algorithms in the HTM which emits an array of probabilities for the inputs it has seen previously. The method you refer to is one which returns the value with the highest probability out of all of the probabilities for that cycle of activity. You can if you like look at each of the probabilities and they can be retrieved as an array - but the method in question is just a convenience for retrieving the one the HTM predicts is most likely. Hope that helps!?

Cheers,
David

1 Like

So, if I understand, for my goal (to check when a value is abnormal) I need just the value returned by .getAnomalyScore(), right?

1 Like

Yes