Using NAB to run experiments on HTM

Engineering NAB
,
1

Hi all,

I think NAB is a great initiative, because it is an objective test problem that produces a quantified score. That gives us a handle. A measure we can use to ask questions, like what is the effect of changing the inhibition algorithm, or how sensitive are the results to the feed-forward connection density, or what happens if we take dynamic time steps, etc. That is, sensitivity analysis.

Really it’s making a statement about what HTMs should be trying to do, and formulating that precisely. Obviously it’s not the whole story, there is much more that HTM should be able to do eventually, but it is something.

Here is an account of what I’ve found so far:
http://floybix.github.io/2016/07/01/attempting-nab

I hope this is useful, and look forward to discussing it.

6 Likes
2

That’s really in-depth, @floybix, thanks. I’m still digesting it, but meantime wondered if paCLA was on in Comportex in your experiment?

3

No it wasn’t, I haven’t tested paCLA with NAB (yet).

Cheers

4

The big news to me is this:

The headline result here comes from Comportex with mostly the same parameters as the Numenta model, but with the following differences:

  • first-order transition memory (1 cell per column, down from 32);
  • a potential receptive field sample of 16% (down from 80%);
  • only 16 segments per cell (down from 128);
  • sampled linear encoders;
  • timestamp given as distal input (not proximal / driving input);
  • a distal stimulus threshold of 18 (not 20).

It makes me wonder if we can improve NuPIC’s NAB scores by doing the same things. @alavin @subutai Do you think this would be a worthwhile experiment?

1 Like
5

That a first-order model gives an improved result suggests that the current design of HTM transition memory is incomplete. While it is possible that my own implementation has subtle problems, the fact remains that HTM’s higher order transition memory has not been demonstrated to have a benefit on any real problem, as far as I know. My feeling is that transition memory will work better when constrained by higher-level contexts, i.e. with temporal pooling.

One thing that’s worth noting about the current “Numenta HTM” detector: it uses the old TP which includes other features like backtracking and “pay attention mode”.

You bring up the fact that high-order sequence memory sacrifices some ability to do first-order sequence memory. Backtracking addresses this problem, though backtracking might not be biological.

This isn’t “the solution”, it’s just worth noting.

1 Like
6

I see. So you would expect that running Numenta HTM detector (old TP) with 1 cell per column would have the same or worse score as 32 cells per column.