Help debugging HTM.java for anomaly detection

@alavin

TM verification done…

Ok, Here’s a GIST containing the identical output for the TMs using the SP output from Python - confirming for another input file, exact output.

This comparison uses: NAB/data/realTraffic/TravelTime_387.csv

I’m now going to work on the SP to confirm my assertions of identical initialization and degrading identical output (my assumption is that it’s due to floating point errors). Stay tuned…

1 Like

@rhyolight

Got side tracked last night with Gradle update issue when trying to use a Snapshot I uploaded to Sonatype especially for your use. Will try and resolve things this morning (after dropping my son off at school).

@rhyolight, take a look at your gitter chat - I left a link and an explanation - basically the code and hotgym jar are available in my fork (master).

With the latest HTM.Java 0.6.9-alpha, the anomaly score divergence with NuPIC is less than 0.6.4-alpha, but still significant. I can’t really make comparisons until ensure the parameters being used are as close to NuPIC’s params as possible.

What NuPIC code/utility running that same file are you using to compare? (I’m assuming you’re running the same file?) I don’t see how anything you might be comparing could possibly be the same unless the exact code you’re running; input file, and properties are the same? This is why I have the QuickTest test code, Matt.

QuickTest compares the same input data in HTM.Java and NuPIC and plots a comparison of their anomaly scores?

Doesn’t do plotting…, but you could output to a file and then plot both?

@rhyolight
EDIT: Here’s a Gist containing both versions (python and java), capable of identically formatted output

These are workhorse files though, Matt. You will have to tweak things you want to change…

Thanks, but I have what I want exactly already with the examples that already exist in your repo and the nupic repo. I did not write any code for this comparison (yet), I simply ran both examples and plotted the results.

1 Like

@rhyolight Would you like me to revert the output format of the HotGym demo?

@alavin, @lscheinkman, @rhyolight
Status:

As promised, here is the proof for the other half of my claim, which is that both the spatial pooler initialization; and output for both languages are exactly the same (initialization being unwaveringly identical) and the output being identical for a similar number of records until it varies.

With the last file, it varied around line 42; this time it varies at line 58 with the input file: NAB/data/realTraffic/TravelTime_387.csv

I only include the first 200 lines of the output because all ~2900 lines aren’t really necessary to see the point at which it varies. But - in addition you also get all the calls to the UniversalRandom RNG and you can see from that that the initialization is exactly the same.

So my claim that the SP is undergoing deviations due to floating point drift seems to be substantiated?

I would be interested to know how close the C++ and Python versions are?

@rhyolight At this point I would maybe like to see if we could arrange another powwow? Because I’m not sure where to go from here? At this point I’m fairly drawn to the conclusion that configurations or something else outside of HTM.Java is causing the difficulty? My questions now are to the NAB author(s?) as to what could possibly be causing these scores? (Also Posted Above)

Running score normalization step
Final score for 'htmjava' detector on 'reward_low_FP_rate' profile = -1.76
Final score for 'htmjava' detector on 'reward_low_FN_rate' profile = 5.28
Final score for 'htmjava' detector on 'standard' profile = 1.46

I think I have done adequate due-diligence on my end and now I’m ready to hand the reigns to Numenta?

@alavin, @lscheinkman, @rhyolight
Status (cont’d):

What I’ve proven so far:

  1. The NAPI causes zero perturbation of the output. I proved this by running the same data file through 2 different configurations of the NAPI (both line by line and Threaded); and in both cases the output is identical to when data is sent through raw algorithms chained together.
  2. The Java components comprising the algorithmic processing are identically compatible. This algorithms in question are: RandomNumberGenerator, TemporalMemory, SpatialPooler, Anomaly (calculator).

Testing Methods:

  • For the RNG (UniversalRandom), outputs are exactly the same between Python and Java, using different interspersed calls (both bounded and unbounded), including calls to sample()

-For the TemporalMemory, the Python ScalarEncoder to SP outputs from 2 different NAB data files, were saved to a file and then run through both the Python and Java versions. Output is exactly the same with absolutely no deviation.

-For the SpatialPooler, 2 raw NAB data files were run through the ScalarEncoder and SpatialPooler of the respective languages, showing identical initialization of SP column pools via checking by eye and monitoring of RNG calls to show exact same output. Then the outputs of both are identical until aprox. line 42 - 58 (depending on input data file) when it is assumed that floating point drift in scaled precision starts to take its toll.

-In addition, there are the identical unit tests for all components involved.

HTM.Java’s TM is now synced to this hash: 6dd098dcc6d333672d4421c3b7bce7df691bfbfc

HTM.Java’s SP is now synced to the PR before @rhyolight’s changes to fix overlaps with no winner columns…

Please comment on other things that can be tested to “vett” the ported algorithms if you feel there is more to be done there?

@alavin
Status:

Added two new methods to UniversalRandom, both python and java - to support the porting of the SpatialPoolerCompatibilityTest.

This will help establish validity of HTM.Java’s SpatialPooler by guaranteeing exact same output as the compatibility test used to validate the C++ SpatialPooler.

My testing method will be to save the output of the Python SpatialPooler during the test and match the exact output with the Java version. With the help of the UniversalRandom, this all becomes possible using the technique used in the compatibility test which is to compare the outputted active columns, and the permanence states, then for exact equality; then copy the permanence states from the Python version to the C++ version (Java version in my case - using the saved output).

So far I have:

  • Identical generation of the random input matrix
  • Identical generation of the initial permanences
  • Identical generation of the input matrix column mapping
  • Identical generation of the initial connected counts for each column’s pool

Next up is to write the procedure for comparing state after each cycle; and then write the procedure for copying the saved Python permanences to the Java proximal dendrites at the end of each cycle (to combat floating point drift).

@alavin @lscheinkman @mrcslws @rhyolight
Status:

As part of the effort to establish long-term verification, this python spatial pooler output writer was created . This script will write 2 Java Class files to a test package inside your HTM.Java repo. The class files are Activations.java and Permanences.java, and contain methods to return 2 dimensional arrays that will be used by the SpatialPoolerCompatilityTest.java unit test to compare activations (sdrs) and permanences at the end of every compute cycle, as well as normalize the permanences every configured update period (i.e. every 10 compute cycles or so…) to set the permanences to combat floating point accuracy drift.

If anyone can take a look and tell me what Pythonisms I’m violating, it would be really appreciated. The file works, but I would like to learn the proper way to do things in the Python world, along the way?

@alavin @lscheinkman @mrcslws @rhyolight
Status:

Success!!!

The compatibility test passes!! Yay! I’ve only created the testCompatability1() method which mirrors the Python test for same. It randomly enables/disables the “learn” parameter for each of the 100 iterations, and I then stored the output as a class file (mentioned above) and then compare the permanences, connected matrix, and activations for each iteration’s record and they are exactly the same! Every 10th iteration I copy the permanences from the Python stored permanences to slightly tweak the column permanences just as the Python compatibility test does for the C++ SpatialPooler. It would be interesting to play with how often that is done.

So now the algorithms and the network api are thoroughly vetted. The answer must now be in the configurations or some other peripheral entity I haven’t thought of? (RandomDistributedScalarEncoder?)

@rhyolight I’m going to check in the new test and support classes…

1 Like

Do we have parity between the SP and TM parameters?

And why do you suggest the RDSE?

That’s next on the list. I have to figure out how to observe them while running cross-process NAB code

I’m surprised this wasn’t first on the list, but that’s neither here nor there. My understanding there is not a 1:1 mapping of parameters from nupic to htm.java. Is this correct? In nupic we specify a dict of params, either in a json or in a .py file. Is there a similar convention in htm.java?

@alavin
Hi Alex,

It was the first thing I looked at, if you recall the earliest conversations with Luiz - and our meeting? However, I moved away from that after I had no immediate success due to the fast moving circumstances in the background where Marcus and @amalta were just getting started on their TM enhancements. I had a two-pronged obligation to keep up with the porting effort and debug the NAB issue - which I felt I could solve all at once by simply ensuring that the port re-writing and testing were held to an extremely rigorous standard (which they indeed were).

Hindsight has now shown that the preponderant probability is that the issue lies elsewhere outside of the algorithms or network - though there were a few issues that got cleaned up along the way; but we couldn’t have known this at the beginning.

Actually, HTM.Java has a very thought-out Parameters system that I’ve been advocating that something similar and Pythonic, be used with NuPIC too. (see the wiki which describes its use, here).

It’s very nice, parameters are configured and then applied via Parameters.apply(<any object>) (It can be any object because they are applied via reflection. For now, they’re only “applied” to a Connections object. The Connections object contains the column/cell matrix and operations to mutate it. Parameters in HTM.Java have bounds, type, and range validation too! As a whole, the Parameters object can be manipulated also. For instance, you can get just ones that apply to a particular algorithm (such as Parameters.getSpatialDefaultParameters()) - or you can get all the defaults - and copy them, or do a union with other Parameters; to combine them with another Parameters object.

In HTM.Java, the algorithms have no state - they are just “actions” applied to a column/cell structure - and so the variables were removed from the algorithms and placed inside the Connections object. This has several advantages, which I’ve been preaching about but I’m not going to go into here. (Having to do with serialization, and ability to employ concurrency, among others…)

At the algorithm level, HTM.Java has every single parameter that NuPIC does - since its a direct port. However, the Network API is different, and also there is no corollary for an OPF - so it does not contain the exact NuPIC OPF parameters that act upon the network as a whole. Those would be the only difference.

My next task is to do a deep dive into the configurations transference and see if I can find any problem there… And too, I haven’t yet run the NAB with the most recent changes - so I have to see what the state of things are.

Actually the reason why I asked for someone to give me the method I can use to do the NAB calculations locally was because debugging across that process divide is ridiculously hard.

I want to run the NAB input file(s) and do calculations locally and then play with the configurations to find the problem. Easy - peasy, but nobody wanted to help! Instead I got a reference to a white paper! I’m just trying to get this done and to be honest, given all the ~2 months effort I’ve put in, I was looking for more enthusiastic compliance with my request because we’re at the finish line now - but no.

Sorry for this emotional communication - but that’s just the way I feel! Compiling the calculations and figuring them out is disproportionately greater in effort for me than it would be to the people who wrote it. Why should I have to to take a week out to shore up my math skills (I’m an EE but its been years since I used any serious math), and scour the document; write the code and then get back to the effort a week or two later - when I can have someone summarize it for me and then allow me to get the debugging step finally done? Especially since this is supposed to be a “team” effort?