Yes, those are the positions of the “1” bits in an SDR representing the concepts which were discovered by Clarifai in the image. The size of the bit array, by default, is 16384 (this is the SDR size used by Cortical.io’s “retina”)
HTM may not be the right tool for this particular problem. There are a couple of potential issues which come to mind:
-
Unless the system were trained on the same episode of the program without advertisements, it would be more likely to flag the TV show as anomalous, since advertisements are repetitive and predictable. It seems unlikely to me that the content of new episodes in a program will share many temporal patterns with previous episodes. On the other hand, this could be used to your advantage, where you train the system only on advertisements, and have a low anomaly score trigger your “commercials” logic.
-
The classifications made by Clarifai are not super granular, so it is unlikely that it would detect simply a background change. Clarifai is classifying objects that it recognizes. That said, concepts in a background might be picked up if there were things like clouds or trees that Clarifai could get a strong hit on.
-
The repeating inputs problem will not allow you to sample every frame and pass it to TM directly. You’d probably need to add some custom code to control when to progress each timestep, and/or implement a function for encoding specific timing.
One approach which comes to mind is to start with simple SDR operations. If the SDR for one frame has low overlap with the SDR from the previous frame, then likely a scene changed. This wouldn’t distinguish between scene changes in a program versus a switch to commercial break though. You could train the system on lots of commercials ahead of time. Then run it with learning disabled. When a scene change is detected, do a reset, then anomaly detection to see if the new scene is one of the advertisements it has been trained to recognize.