Great! Thanks a lot, I saw that you had done this earlier!
Cheers,
David
Thank you for the explanation @cogmission, Iâll take a look tonight when I have time.
Now that youâve corrected this, have you tried running the detector on NAB?
Alex,
Thanks, having the additions to that file will help me analyze the problem. When I said I wasnât using the âpredictiveColumnsâ I was referring to the âcontrolâ file not the NAPI. The NAPI was doing it right from the beginning. This confirmed that the NAPI was doing everything correctly and just like the âcontrolâ file which was a simple chain of algorithms.
Therefore, the NAB problem lies elsewhere. Thatâs why I want a simplistic, officially vetted Python version of the chain of algorithms so I can compare the entire thing because maybe there is a problem in one of the algorithms, or there is a problem in the NAB detector. But I need your inclusions in the Python example file to begin to narrow things down.
Thanks for your help Alex!
Cheers,
David
Any word on adding the Anomaly calculation to that file? Who is actually going to do this, Alex you or Matt you? 
I donât want to rush anyone, I just want to know whatâs going on? I know both of you are stretched thin and very busy - but I just want to know whatâs happening thatâs all?
@fergalbyrne Iâm going to post (edit) the gist with the comparison code so you can see thereâs no discrepancy in the NAPI. To summarize to be clear: running the data through raw algorithms and the NAPI produces exactly the same output so we can consider that proof that the NAPI doesnât impact the process - though there may be a problem elsewhere (in the original Java Anomaly code, or in the detector). If youâve been following this, I am going to use a bare bones Python version of my test code to compare and see where any discrepancies lie. Given this information, let me know from what angle you want to continue your investigation, if at all? Also clean up of the code is still very welcome if thatâs all you want to do?
Link to most recent Gist revision
Thanks guys!
David
Like I said earlier, I canât get to this until next week. I can probably look into it then. I need to write some code against the NuPIC Network API anyway, so this will a good chance.
1 Like
Thank you Matt. I didnât mean to rush you its just that Alex wrote after you said this that he would look at it that night⌠So I got confused as to who was going to do it and when. No problem, I appreciate your help! 
1 Like
I plan on taking a look at this, but havenât had a chance to come up for air this week. Hopefully later today 
1 Like
Ok, you guys can duke it out in the corner⌠just donât hold me responsible!
No really, itâs fine whoever can get to it and when⌠I really appreciate it guys!
Hey Guys,
I found a bug that could be very well be causing the errors. I added a more stringent test to the test class which I said had identical output. Instead of merely outputting the results as they are being produced, I collected a sample from both the control code and the NAPI code. I then did a DeepEquals on the Sample and discovered that the activeColumns (the feedForwardActiveColumns field) were later overwritten! As it turns out, that variable in the Layer class that was being used internally to set a value in the ongoing calculations!
So thanks to this rigid test, I was able to catch it and fix it. Now taking samples from both the control and the NAPI generated output now leads to identical (non-overwritten) output. This could have been the problem and was obscured because immediate printout was clean.
I now submitted a PR and added the NetworkConsistencyTest which demonstrates all fields are now isolated and nothing overwritten and no âre-used pointersâ are occurring.
See here:
@lscheinkman or @alavin if you could update your HTM.Java clones and test this, it would be appreciated. I want to see if the NAB runs better with this fix. I canât run the NAB yet until NuPIC is building again and it will still take me some time because I have to reinstall NuPIC to get the latest code. So if one of you could run this, Iâd appreciate it, if you have time?
Cheers,
David
2 Likes
@cogmission
I ran NAB with the latest âhtm.javaâ from master (e3c50a2) but unfortunately I got the same results.
Final score for 'htmjava' detector on 'reward_low_FP_rate' profile = 3.74
Final score for 'htmjava' detector on 'reward_low_FN_rate' profile = 12.80
Final score for 'htmjava' detector on 'standard' profile = 9.29
Luiz
1 Like
Ok, weâll keep digging!
@cogmission @rhyolight Iâm having issues pushing to the gist, so hereâs the reworked script:
'''
Created on Feb 8, 2015
@author: David Ray
'''
import numpy as np
import pprint
from nupic.frameworks.opf.common_models.cluster_params import (
getScalarMetricWithTimeOfDayAnomalyParams)
from nupic.frameworks.opf.modelfactory import ModelFactory
class Layer():
""" Makeshift Layer to contain and operate on algorithmic entities """
def __init__(self, networkInstance):
self.networkInstance = networkInstance
self.sensor = self.networkInstance._getSensorRegion().getSelf()
self.sp = self.networkInstance._getSPRegion().getSelf()
self.tm = self.networkInstance._getTPRegion()
self.tm.getSelf().computePredictedActiveCellIndices = True
self.theNum = 0
def input(self, value, recordNum, sequenceNum):
""" Feed the incremented input into the Layer components """
if recordNum == 1:
recordOut = "Monday (1)"
elif recordNum == 2:
recordOut = "Tuesday (2)"
elif recordNum == 3:
recordOut = "Wednesday (3)"
elif recordNum == 4:
recordOut = "Thursday (4)"
elif recordNum == 5:
recordOut = "Friday (5)"
elif recordNum == 6:
recordOut = "Saturday (6)"
else: recordOut = "Sunday (7)"
if recordNum == 1:
self.theNum += 1
if self.theNum == 100:
print "bl"
print "--------------------------------------------------------"
print "Iteration: " + str(self.theNum)
print "===== " + str(recordOut) + " - Sequence Num: " + str(sequenceNum) + " ====="
output = np.zeros(self.sp.columnDimensions)
# Run through network model
inputData = {"value": value}
result = self.networkInstance.run(inputData)
rawScore = result.inferences["anomalyScore"]
# Print out some info for the...
# ... encoder
print "RDSEncoder Input = ", value
print "RDSEncoder Output = "
print "\t", self.sensor.getOutputValues('sourceEncodings')[0].nonzero()[0]
# ... spatial pooler
print "SpatialPooler Output = "
print "\t", self.sp._spatialPoolerOutput.nonzero()[0]
# ... temporal memory
print "TemporalMemory Output (active cells) = "
print "\t", self.tm.getOutputData("bottomUpOut").nonzero()[0]
print "TemporalMemory correct predictions (active cells that were previously predicted) = "
print "\t", self.tm.getOutputData('predictedActiveCells').nonzero()[0]
def _createNetwork(minVal, maxVal, verbosity=1):
# Create model
modelParams = getScalarMetricWithTimeOfDayAnomalyParams(
metricData = np.array(()),
minVal=minVal,
maxVal=maxVal,
tmImplementation = "cpp"
)["modelConfig"]
if verbosity > 0:
print "Model params:"
pprint.pprint(modelParams)
# Setup encoder params for this test data
__setupEncoder(modelParams["modelParams"]["sensorParams"]["encoders"])
model = ModelFactory.create(modelParams)
model.enableInference({"predictedField": "value"})
return model
def __setupEncoder(encoderParams):
encoderParams.pop("c0_dayOfWeek")
encoderParams.pop("c0_timeOfDay")
encoderParams.pop("c0_weekend")
encoderParams["timestamp_dayOfWeek"] = None
encoderParams["timestamp_timeOfDay"] = None
encoderParams["timestamp_weekend"] = None
encoderParams["value"] = encoderParams.pop("c1")
encoderParams["value"]["fieldname"] = "value"
encoderParams["value"]["name"] = "value"
def _runThroughLayer(layer, recordNum, sequenceNum):
layer.input(recordNum, recordNum, sequenceNum) # value = recordNum ???
if __name__ == '__main__':
# Create a network model that expects metric data in the range 1,7.
net = _createNetwork(1, 7, verbosity=1)
layer = Layer(net)
i = 1
for x in xrange(2000):
if i == 1:
layer.networkInstance.resetSequenceStates()
_runThroughLayer(layer, i, x)
i = 1 if i == 7 else i + 1
2 Likes
Thank you, Iâll work on it first thing tomorrow!
Hi Alex,
First, let me say, âThank Youâ, for all of your hard work!
The idea was to avoid container constructs like Model and ModelFactory and to show what is going in and out of each algorithm (no support frameworks). So to start around line 84 in the QuickTest.py file and physically add the necessary code to feed the output of the TemporalMemory in to the Anomaly code and provide printouts of the input data and output data for same?
We want to see:
- What exactly goes into the Anomaly code and exactly what comes out of it?
- What transformations of the data are necessary (if any) to take the output from the TemporalMemory and pass it into the Anomaly code?
- The structure kept the same as the original file so that it is the mirror image of the QuickTest.java file. (I apologize, you didnât know that existed).
- So if possible, on line 12, please add the direct import of the Anomaly code; on line 24, please declare the reference variable to the Anomaly code, and on line 85 please begin transforming the TM output (with printouts) and add it into the Anomaly class keeping the same style of documentation - and printout the Anomaly result?
- Nothing hidden, everything simple and inlined right in front of us so we can see exactly whatâs going on⌠(Please donât include anything that requires debugging into another file that isnât the Anomaly code itself?)
Please keep in mind. This is âHTMNetwork for Dummiesâ
Another point is. If the file isnât exactly like whatâs mentioned, then I canât make a Java file that does the same simple thing and I canât compare everything minutely enough for trouble shooting our issue. I guess I failed to mention there is a QuickTest.java file as well and it looks almost exactly the same and has exactly the same output. Also we donât want to get rid of the Classifier code, we want to output everything so we can see whatâs going in and out of everything.
I really apologize, I know your time is valuable but this code has to act as a âsanity checkâ for seeing right in front of my face, what goes into and comes right out of each and every algorithm so that I can apply that knowledge to HTM.Java.
In addition, this is the file I point to (in both languages) when people want the most accessible example of how to work with the algorithms themselves.
Anyway, please let me know if my craziness requires any more explanation? And thanks a million for letting me pester you with these exact requirements, again I know how valuable your time is and I really appreciate everything youâre doing to help debug things.
Cheers,
David
1 Like
@cogmission, thereâs nothing happening in the Anomaly code except itâs doing the calculation of the rawScore, which is just (1 - predicted/active) - that is the correct calculation. The problem is (or was if the TM/Observer code is now fixed) that the wrong information is/was coming out of the TM/Observer to feed that calculation.
1 Like
I donât believe at least, that it is fixed just yet because @lscheinkman tested my changes - but as soon as @alavin finishes the Python simple layer file, Iâm going to test it against the Java version which should get similar quality results, and if that works Iâm going to test it against the NAPI and see where it goes wrong⌠Those are my plans for now, donât know how else to approach thisâŚ
Whatâs coming out of the TM Observer is the same as whatâs coming out of the TM by itself, exactly! Thatâs the problem!
1 Like
I donât know. I suggest you test that hypothesis:
- Run the TM/Observer over a dataset and record its outputs.
- Donât connect the Observer and just record the TM outputs on the same data (with the same RNG seed).
- Compare.
If they donât match exactly, then the attachment of the Observer is causing mutation in the TM, and creating erroneous output.
edit: The outputs of interest to the NAB issue are previousPredictedColumns and activeColumns.
2 Likes
I did exactly that! Did you not read over the earlier messages? I even posted the Gist code that proves it?
https://gist.github.com/cogmission/1b8386700368e37e8764c4b39caff55d
Itâs now proven ⌠So I donât what to doâŚ? But weâll push onâŚ
Eh, no. Look at the NAB code. It doesnât use Anomaly.compute(). It accesses fields of the Observer, and runs methods on the Observer. Youâre not testing the same thing. And the NAB code uses a standard Layer, not this SimpleLayer which doesnât run the same methods.
As Iâve been saying all along, I suspect that the Observer and TM are interacting in some way. In the computation actually being run by the NAB code. Why donât you just run the NAB code with and without the Observer there at all, and compare what comes out? The NAB runner doesnât need to use the Observer, because it feeds the HTM the inputs one at a time.
Iâm sorry buddy, but I donât believe thatâs correct. The NAB interacts with HTM.Java code through the HTMModel that bridges the two (that @lscheinkman) wrote.
Unless there is something Iâm missing? @lscheinkman/@alavin ? Can you please confirm from where the NAB is getting its data?
Also, I canât run the NAB because I have to re-install and Update NuPIC (Yesterday I was waiting on the build to be fixed).
Are you still investigating this? Are you able to run the NAB?