Help debugging HTM.java for anomaly detection

Status Update:

Using the new UniversalRandom RNG, I have finished re-write of the TM and have completed 13 tests (so far) - all with exactly the same output (same cell, segment and synapse indexes chosen from burst etc.)…

I think I’m about half way through the tests…

I’ve read every word :wink:. This project is important to me.

Keep in mind when comparing the anomaly scores you don’t want to use the anomaly likelihood calculation. In NAB this corresponds to the “raw scores”.

Thank you for the update, keep it going!

1 Like

No worries. I know this. It’s just that there’s so much here and I think I misspoke because some of the other stuff I mentioned is actually in other threads, I believe and not here. I really appreciate your support Alex!

Cheers

@alavin

TemporalMemory & Test re-write complete! :confetti_ball:

Next step: Assemble @rhyolight’s handy work with yours and my own to make rudimentary layer in both Python and Java and get back to testing NAB! (probably starting tomorrow because I’ve already been working 13 hours [since 1:00am])… Just a heads-up! (I’ll check the code in for merging tomorrow, I have to remove the old stuff and make sure the TemporalMemoryMonitorMixin stuff still works)

1 Like

@alavin @rhyolight @fergalbyrne @lscheinkman @mrcslws

Status:

Upon removal of the “old” TemporalMemory, I have more than a few tests to examine; either for fixing/adjustment, removal or rewrite - depending on the nature of the failure. This is to say that the anticipated step of swapping the old for the new is a bit more involved than previously expected.

My current goal is for the new code to be error free by Monday, and to start back in on preparing the skeleton code to be able to return to NAB testing then.

Cheers,
David

1 Like

A post was split to a new topic: Potential bug in spatial pooler

@alavin

Hi I just want to make sure exactly what is supposed to go into the Anomaly.computeRawAnomalyScore() method? I can’t believe I’m asking this again, but this is all so confusing… :stuck_out_tongue:

I’m pretty sure Subutai said that you take:

  1. Take the output of the SpatialPooler which represents the “activeColumns” coming in to the TemporalMemory.
  2. The previously predictive columns (as derived from the predictive Cells in t - 1)
  3. So the comparison is between the predicted input of the TM in t - 1 and the current input coming from the SP?

But now I’m not sure if you take the currently activated columns from the TM and not the TM’s input?

Also above you probably meant “columns” but you said “cells” and I just want to make sure?

So I just need to clear this up?

Sorry for the delay @cogmission, on PTO…

To reiterate this:

	for each timestep t: 
		1. run data through encoder
		2. run SP computation
		3. get values for raw anomaly score calculation:
			a = representation of input at time t (i.e. SP currently active columns)
			b = representation of prediction at time t-1 (i.e. columns of TM's currently predicted cells)
		4. compute raw anomaly score:
			rawScore = 1 - b/a
		5. run TM computation
1 Like

@alavin,

Sorry for contacting you on your PTO (Matt also informed me that you were away until the end of the week)… …and thank you for responding!

That clears that up. I just wanted to make sure.

Hi David, any updates on this?

@alavin

Thank you for keeping up on things! Well, I’ve (prepared/am finishing up) a test harness which I can have run any combination of algorithms in a chain (Python and Java matching versions). The current version of that confirms that after 50 or so calls to the UniversalRandom during TM intialization (using the most recent re-written TM and not the final one), and processing of one row, produces the same output and the same 50 or so random integers when running both the Python and Java version.

I’m waiting on https://github.com/numenta/nupic/pull/3254 to be merged, and then I’m going to re-write the Java version to do the same thing. After that, I’ll begin testing everything (very very minutely) in earnest and I expect to find what divergence exists (if any does after the TM re-write). Also I’ve created another branch to do an SP update of the Java version to match the Python SP code - but that will be done after making sure the Java version gets good Anomaly results - then I will create a new PR for a new SP.

@cogmission, https://github.com/numenta/nupic/pull/3254 was just merged.

2 Likes

@amalta Good Job!
I know! I’m so excited! Ok, well I’ve got to update my fork and get to work! :wink:

@rhyolight
I update my fork, then run:
python setup.py develop --user
right?

Yeah that should work. Not sure if you care, but if you want to see the compatibility test pass, you would have to update your nupic.core fork then compile/install the bindings.

Thanks, but I think I’ll wait to update nupic.core, it required major surgery last time - I’m sure I’ll need @rhyolight’s assistance for that…

1 Like

oh god please no

3 Likes

@alavin
Status Update:

After studying the itertools.groupby() and the new groupby2() code, and dissecting it’s output at various stages; I’ve created a Java equivalent: A GroupBy() class that is used by another GroupBy2() class and has the same algorithmic treatment and output as confirmed by the two new tests, GroupByTest(); and GroupBy2Test() which perform the same tests and yield the similar result data containers with the exact same output as their Python counterparts. The two new classes can be used in the same way in the same call site within the new TemporalMemory (not as yet written).

Today I’m starting in on the new TemporalMemory and TemporalMemoryTest; I expect to be done around Monday. After that, I expect to return to debugging the NAB.

@mrcslws @amalta
Just an fyi point. I altered the exit condition of the GroupBy2 to be triggered by a custom PeekIterator that can query the Generator to see if there’s a match for the new/next key - instead of letting the code throw an exception (and executing a controlled crash), to indicate generator exhaustion. I’m not sure how the algorithm performs in C++, but something similar may be able to be accomplished in Python? I’m not sure… What I do is:

  • Upon creation of the Iterator, the first entry is processed and saved to a variable nextVar.
  • Then on each call to next() of the iterator, the iterator copies the nextVar to currVar.
  • Then the iterator actually processes the n + 1 entry and then saves it to nextVar - then returns the contents of currVar.
  • Then there is a new method on the Generator called peek() which returns the contents stored in nextVar which would be the data returned upon a subsequent call to next() on the iterator in t+1.
  • The variable nextVar is then set to “null” when there are no more entries to process, and the code can check it to see if the generator is exhausted.

@alavin

Finished re-write of TemporalMemory & test for same! I have some integration tests (the MonitorMixin stuff) to fix, and then I’m back to NAB debugging…

1 Like

Awesome!

1 Like

@alavin

Status Report:

Using the version of the TM (with SegmentOverlaps - the one before @mrcslws’ last merge) we have identical outputs until record 45; so that is where my investigation begins today…

The testing method I used was to run the test file through the Python SpatialPooler and save its output. Then I read in that SpatialPooler output and run it through the Java TemporalMemory. This way I can guarantee identical inputs to both the Python and Java TemporalMemorys.

I’ve included GISTS (only the first 100 lines) of both the Python and Java output which shows:

  1. The Record #
  2. The raw input (timestamp, temperature)
  3. The GroupBy2 Column/Segment Generator output
  4. The calls to the UniversalRandom next() method. Next is called with the bounds in parenthesis i.e. next(bound) <-- Even the bounds of the RNG calls are the same!
  5. The TM input (Sparse output of SP)
  6. The TM prediction
  7. The TM active columns (derived from the active cells)

For all of these you will see EXACT output duplication. Even the line numbers of every single output and their format, are identical (Thanks to the UniversalRandom RNG)

So now I’m investigating Record #45 to see why the Java version makes an additional call to the RNG and diverges on that record.

Right now I’m confident the Java NAB “poor results issue” has been fixed, but I just want to take it all the way and get identical output. Also, I need to run things through the Java SpatialPooler and ensure good results. Exact output using the Java SpatialPooler will have to come later because there are one or two “exotic” RNG methods being used by the Python SP. (I already equalized the “sample()” method to be the same in the UniversalRandom, but there may be another method I need to “normalize”).

It shouldn’t take me any more than a day or two to reach my goal of identical TM output, after which I’m not going to play with it anymore (after porting @mrcslws ’ latest merge) - and I’ll run the NAB to verify good results (I already see from the quality of the anomaly scores that the results will be in line with NuPIC, but like I said, I want to find the reason for the slight divergence.)

1 Like