@cogmission the script you seek is here: https://github.com/numenta/nupic.research/blob/master/htmresearch/algorithms/anomaly_detection/numentaTM_low_level.py (thank you @mrcslws).
For instructions on how to run this, refer to the readme at https://github.com/numenta/nupic.research/tree/master/projects/nab_experiments, particularly the second invocation for running specific NAB data files.
@alavin I saw you guys working on that - thank you so much!
Matt advised me to get some down time. I think we could all agree itās sorely needed 
So Iām at the movies with my son, and will try this out later tonight or tomorrow morning.
You guys are great, thanks again!
Cheers,
David
Guys - thank you very much for this!
Does NAB use the likelihood functionality or does it use just the raw scores? Any chance I can just get this like Iāve been looking at it with the raw anomaly calculation so I have as little complexity as possible to debug? Then maybe later we can get into the likelihood stuff, but I donāt want to have to deal with that just to vet the difference between HTM.Java and NuPIC ok?
EDIT: Sorry I saw the likelihood stuff first and wrote this without looking in the handleRecord methodā¦
@rhyolight this doesnāt get you off the hook though 
LOL! because I canāt do a direct thing-for-thing comparison using NAB because the Java detector is impossible to trouble shoot due to its cross-process natureā¦ So I still need something simple that I can run side by side in Eclipse and really see whatās going on in a direct comparisonā¦
Thanks,
David
Ok I got Marcusā script to run like this:
python run.py -d numentaTMLowLevel --dataDir ~/git/NAB/data --windowsFile ~/git/NAB/labels/combined_windows_tiny.json --profilesFile ~/git/NAB/config/profiles.json --detect
But when I try to run the htmjava detector so that I can see the same output side by side, I get:
Traceback (most recent call last):
File ārun.pyā, line 183, in
main(args)
File ārun.pyā, line 68, in main
detectorConstructors = getDetectorClassConstructors(args.detectors)
File ārun.pyā, line 41, in getDetectorClassConstructors
d : globals()[detectorNameToClass(d)] for d in detectors}
File ārun.pyā, line 41, in
d : globals()[detectorNameToClass(d)] for d in detectors}
KeyError: āHtmjavaDetectorā
Iām assuming it canāt find the htmjava detector due to some kind of path error or something?
I have only have Luizā branch checked out as NAB, so any NAB directory lookup will find the version with Luizā branch which includes the htmjava detector. It resides at: /Users/cogmission/git/NAB which you can see from the above python run command, is the default relative directory for NAB.
Hereās a pic to answer any name resolution questionsā¦
I know whatās up here. Iāll send you a reply later this morning when I have time. What command are you doing to run htm java?
I forgot to mention this is the command I tried to use to run the htmjava detector:
python run.py -d htmjava --dataDir ~/git/NAB/data --windowsFile ~/git/NAB/labels/combined_windows_tiny.json --profilesFile ~/git/NAB/config/profiles.json --detect
Thanks Alex!
Thanks David. And where do you have the script ā where are you running run.py from?
Hereās what to run @cogmission:
cd ~/git/NAB
python run.py -d htmjava --detect --windowsFile labels/combined_windows_tiny.json
# This runs the HTM.java detector in "~/git/NAB/detectors/htmjava/" on the subset of data specified in the windowsFile json.
cd <your path to nupic.research/projects/nab_experiments>
python run.py -d numentaTMLowLevel --detect --dataDir ~/git/NAB/data --windowsFile ~/git/NAB/labels/combined_windows_tiny.json --profilesFile ~/git/NAB/config/profiles.json
# This runs the NuPIC TM detector in "<path to nupic.research/htmresearch/algorithms/anomaly_detection/numentaTM_low_level.py>" on the subset of data specified in the windowsFile json.
Comparing the two outputs, what strikes me immediately is the lock-step synchronicity in the āreactionsā of both Anomaly detectors. They both have an anomaly of 1.0 until record 11, then at record 11 they both register ālessā of an anomalyā¦
NuPIC == 0.825
htmjava == 0.25
I think whatās going on here is that htmjavaās RDSE is not getting initialized with the same parameters. It may not be that the āNODEā isnāt getting read like I thought, but that the variables and setup are not aligning such that theyāre getting applied correctly - either due to naming mismatches or a mismatch in how the htmjava RDSE is expecting its configs.
What I see is a lack of gradations in the āresolutionā of the detections. Htmjava seems to swing more āintenselyā between 1.0 and 0.0 where NuPIC seems to have āfinerā differences.
I would say that things are working but that the resolution is somehow not getting set because the changes are in lockstep synchronicity they are just vastly different.
These are just my initial observationsā¦
I posted these files, please let me know what you think?:
The anomaly score column seems to have the same āanomaly_scoreā value until line #606 where they both change to another value for the first timeā¦
They both have: 0.0301029996659 until line #606 when they both change to different values but not the same value.
It would be best to focus on the raw scores, not the āanomaly_scoreā column, leaving the anomaly likelihood calculation out of it.
Yes, but what can we deduce from the comparison of these outputs? What insights can Numenta give me about how the Java TM is performing, possible insights into configuration differences? Anything?
This is Numentaās territory, not mine - have at it.
Also, from the meeting you or Luiz were going to look into what the actual configurations are that HTM.Java has (the RDSE and the other params) - whether they are actually being set on the RDSE? I donāt believe they are because the variable names donāt match up. But Iām at a loss for how to debug this since I can only run it directly and canāt debug across the python runner and itās pre-scanning of the data and passing the min/max values to the Java side. I really think this should be happening on the Java side so we can debug it and not hidden away in the Python so we canāt actually manipulate the JSON node preparation and the scanning of the data to derive the min/max values.
I at least determined that the direct loading of HTMModel is broken (because it in fact loads from a file and not a pre-manicured JSON node) so Luiz needs to guarantee that both logic paths load the same way so that we can see what 's going onā¦ This is why I mistakenly thought that the configs were never getting read - because that Python path of logic loads a different way then the path which loads the config from a file.
What I mean to say is that Iām interested in your analysis of the output in terms of how the configuration might affect the differences?
Mainly because I see things changing in lock-step synchronization which indicated to me that there is no algorithmic or network-compositional problem - and nudges me toward some form of misconfiguration as a likely cause?
@alavin @lscheinkman @rhyolight,
I was able to printout the actual properties used by the Java RDSE during exectutionā¦
Does this look right?
minIndex: 500
maxIndex: 500
w: 21
n: 400
resolution: 1.5825004536384615
offset: null
numTries: 0
name: [1.5825004536384615]
buckets :
[ 500 ]: [194, 38, 160, 390, 151, 84, 104, 340, 243, 255, 159, 358, 101, 355, 249, 14, 100, 19, 224, 369, 269]
Hereās the comparison of JSON parameters passed in, to HTM.Javaās parametersā¦
It does look like theyāre getting set - so the only outstanding question is the RDSE above?
JSON Params passed in:
{
"aggregationInfo":{
"seconds":0,
"fields":[
],
"months":0,
"days":0,
"years":0,
"hours":0,
"microseconds":0,
"weeks":0,
"minutes":0,
"milliseconds":0
},
"model":"CLA",
"version":1,
"predictAheadTime":null,
"modelParams":{
"sensorParams":{
"sensorAutoReset":null,
"encoders":{
"value":{
"name":"value",
"resolution":1.5825004536384615,
"seed":42,
"fieldname":"value",
"type":"RandomDistributedScalarEncoder"
},
"timestamp_dayOfWeek":null,
"timestamp_timeOfDay":{
"fieldname":"timestamp",
"timeOfDay":[
21,
9.49
],
"type":"DateEncoder",
"name":"timestamp"
},
"timestamp_weekend":null
},
"verbosity":0
},
"anomalyParams":{
"anomalyCacheRecords":null,
"autoDetectThreshold":null,
"autoDetectWaitRecords":5030
},
"spParams":{
"columnCount":2048,
"synPermInactiveDec":5.0E-4,
"maxBoost":1.0,
"spatialImp":"cpp",
"inputWidth":0,
"spVerbosity":0,
"synPermConnected":0.2,
"synPermActiveInc":0.003,
"potentialPct":0.8,
"numActiveColumnsPerInhArea":40,
"globalInhibition":1,
"seed":1956
},
"trainSPNetOnlyIfRequested":false,
"clParams":{
"alpha":0.035828933612158,
"verbosity":0,
"steps":"1",
"regionName":"SDRClassifierRegion"
},
"tpParams":{
"columnCount":2048,
"activationThreshold":20,
"cellsPerColumn":32,
"permanenceDec":0.008,
"minThreshold":13,
"inputWidth":2048,
"maxSynapsesPerSegment":128,
"outputType":"normal",
"initialPerm":0.24,
"globalDecay":0.0,
"maxAge":0,
"newSynapseCount":31,
"maxSegmentsPerCell":128,
"permanenceInc":0.04,
"temporalImp":"tm_cpp",
"seed":1960,
"verbosity":0,
"predictedSegmentDecrement":0.001
},
"clEnable":false,
"spEnable":true,
"inferenceType":"TemporalAnomaly",
"tpEnable":true
}
}
Printout of HTM.Java Connections
{
Spatial: {
learn:true
inputDimensions:[454]
potentialRadius:16
potentialPct:0.8
globalInhibition:true
inhibitionRadius:0
localAreaDensity:-1.0
numActiveColumnsPerInhArea:40.0
stimulusThreshold:0.0
synPermInactiveDec:5.0E-4
synPermActiveInc:0.003
synPermConnected:0.2
synPermBelowStimulusInc:0.01
synPermTrimThreshold:0.05
minPctOverlapDutyCycles:0.001
minPctActiveDutyCycles:0.001
dutyCyclePeriod:1000
maxBoost:1.0
spVerbosity:0
}
Temporal: {
columnDimensions:[2048]
cellsPerColumn:32
activationThreshold:20
learningRadius:2048
minThreshold:13
maxNewSynapseCount:31
initialPermanence:0.24
connectedPermanence:0.5
permanenceIncrement:0.04
permanenceDecrement:0.008
predictedSegmentDecrement:0.001
tmVerbosity:0
}
Other: {
random:org.numenta.nupic.util.MersenneTwister@e50a6f6
fieldEncodings:{value={fieldName=value, fieldType=float, resolution=1.5825004536384615, encoderType=RandomDistributedScalarEncoder}, timestamp={fieldName=timestamp, formatPattern=YYYY-MM-dd HH:mm:ss, fieldType=datetime, encoderType=DateEncoder, timeOfDay='21':9.49}}
hasClassifiers:false
}
}
@alavin @lscheinkman @rhyolight
Question:
Is this alright to do?
if(t1.getFeedForwardSparseActives() == null || t1.getPreviousPredictiveCells() == null) {
return t1.anomalyScore(1.0);
}
ā¦or should this be some other return value if there arenāt any predictions ?
Hey,
I updated the script myself. Can you guys tell me if Iām making any Pythony mistakes here?: (check out the few lines at #'s 73 to 79)
Thanksā¦
@cogmission Iāll take a look later this morningā¦
Pythonic issues aside, here are some problems with the script:
lines 62 and 64 should use self.encoder:
62: encoding = self.encoder.encode(value)
64: bucketIdx = self.encoder.getBucketIndices(value)[0]
The TM compute step should come after the anomaly score calculation. We want to compare the regionās representation of the current value (i.e. activeCols) against what the region predicted from the last timestep (i.e. prevPredictedColumns). I donāt see you doing this. This is fundamental to how anomaly detection with HTM works.
Why do anything with this āanomaly computerā? Your lines 71-86 should == https://github.com/numenta/nupic.research/blob/master/htmresearch/algorithms/anomaly_detection/numentaTM_low_level.py#L139-L145.
Okay one Pythonic thing 
: line 71 is odd, better to do activesCols = set(self.spOutput.nonzero()[0].tolist())
@cogmission David, you asked me to update your QuickTest.py script so that it reads its input from one of the NAB scalar anomaly files, so I did that here:
I updated some of the params, and added a date encoder. Also, I am not sure how to hook up the classifier, so I left that part commented out.
Wait! What What?
Yes that totally makes sense! In HTM.Java I store the previous prediction - I got confused here it seems.