Neuromorphic Architecture for the Hierarchical Temporal Memory

[rhyolight]

[rhyolight]

From one of the authors of A Mathematical Formalization of Hierarchical Temporal Memory's Spatial Pooler.

[michaelklachko]

To put this into perspective, 91% on MNIST is an extremely bad result. In fact, just last night I was testing a new architecture from a recent paper, and I couln’t get MNIST accuracy above 97.5%. This pretty much was a show stopper. An acceptable result would be at least 99%.

[gmirey]

MNIST recognition ain’t precisely a good fit for HTM, but we didn’t need a neuromorphic architecture to know that.
So, what I see as interesting here is the 1364x speedup

however we poor implenters-in-a-garage won’t benefit from such things for a while :x

[dskinner]

Pardon the newbie question, but can you explain why you think MNIST is not a good fit for HTM? Are you aware of anyone building an SDR encoder for image learning/classification in general, perhaps using ImageNet or CIFAR datasets?

[dskinner]

From what I can find, the human error rate for classifying MNIST is 2% - 2.5%. Can you explain why you think that a HTM version that achieves human-level accuracy is a show stopper? Isn’t it just as likely that for a simple dataset like MNIST that the neural network solutions are overfitting?

[gmirey]

Current HTM theory is built around a single level dealing with temporal sequences, and may detect unexpected elements when a sequence has started to match.

Recognition of objects, on the other hand, would in my view involve a hierarchical model. Neural networks succeeding at MNIST classification with very high rates typically are deep (many layers in a hierarchy). Assuming the problem of encoding a visual image as binary SDRs has been solved, a single ‘spatial pooler’ would only get you so far, and thus an HTM based solution to image recognition would also need a hierarchy.

Some people here are trying to tweak the current ‘single-level’ HTM to a model supporting a hierarchy, but this is not part of current Numenta implementation, and is as far as I know an open problem.

[dskinner]

Thanks for the explanation. I remember in the HTM videos that hierarchy was discussed, and since the output of the pooler is another SDR it seems that a hierarchical system is certainly possible, so are you saying simply that Numenta hasn’t “yet” implemented a hierarchical solution in code, or that there’s a more fundamental part of the model that is missing (and thus preventing implementation)?

[Bitking]

H is for Hierarchy, but where is it?

Continuing the discussion from The art of SDRs: Some of you might be thinking… Hey, you call this stuff HTM but there’s no Hierarchy? What gives? The software architecture for hierarchies exists within both HTM.Java and NuPIC (and Comportex, right @floybix?). You can create models and link them together into a hierarchy, with lower levels passing data up into higher levels. The algorithmic mechanism for creating an effective learned spatiotemporal hierarchy using one layer of an HTM is sti…

Rethinking Hierarchy

I think the point we’re trying to make is this: objects are represented in all levels of the hierarchy. If you think of a coffee cup, you can imagine how it feels in your hands, against your lips. You can feel whether it is smooth or course. You also know it is a container used to transport liquid, usually into human mouths. This information is not present the lower sensory levels of the object’s representation, because they are focused on the small details, like textures or lines and curves. T…

Perhaps @rhyolight has more to add to this.

[michaelklachko]

I believe that human result came from this paper and it was not done on MNIST, but on a smaller US POST dataset, which was considered to be harder than NIST. So it’s not really clear what would be a human performance on MNIST, but I’d say there are only a few dozen of truly unrecognizable digits (to a human). In any case, HTM is 91%, so it’s not even close to 97.5% “human performance”.

I’m not sure if you can say “it’s overfitting”: it’s tested on the previously unseen test set. Actually I agree with others who say that HTM is poorly suited to image recognition. It’s just not very impressive to see these results published as an achievement.

[OS_C]

If some of the digits are unrecognizable to a human, then providing them with a label is basically arbitrary.

A human should have practically perfect performance on something like MNIST, and whatever errors there are would be due to ambiguity, and should in reality be classified as being both of the digits such ambiguously represent. Arbitrarily disambiguating and penalizing as a result of such arbitrary labelling is nonsense.

[acatovic]

Couldn’t agree with you more, I mean try categorising these guys: http://neuralnetworksanddeeplearning.com/images/mnist_really_bad_images.png

[nbro]

@acatovic

Couldn’t agree with you more, I mean try categorising these guys: http://neuralnetworksanddeeplearning.com/images/mnist_really_bad_images.png

Some of them are quite easy (at least, personally) to categorize! I didn’t take more than 5 seconds to categorize each of them separately. (Note: normally, you would classify or identify a “decently” written digit in less than a second). Nonetheless, I agree, there are ambiguous pictures. Furthermore, we aren’t able to categorize some of them because they really aren’t digits anymore (to us).

[acatovic]

Wow you’re good! I don’t think I could do any. I haven’t seen the labels for them, but just looking at them it would be pure guess work for me.

[nbro]

@acatovic

I haven’t seen the labels for them, but just looking at them it would be pure guess work for me.

Then you probably haven’t seen as many handwritten digits as I did. Anyway, again, some of them could be a 4 or 1 (e.g.), but this is actually a common case!

[michaelklachko]

Sure, but you can judge the performance of your model by looking at misclassified images. Ignore all images that are unrecognizable/mislabeled, and make sure there are no easy examples the model has missed. My point was that if a network gets 250 examples wrong (as is the case for 97.5% accuracy), then it definitely got many easy examples wrong.

To provide some more perspective, state of the art on CIFAR-10 is about 98.5% (CIFAR is a much harder dataset), and ~85% on ImageNet (which has 1000 classes, rather than 10).

[OS_C]

True it helps judge performance, but a human eye on errors made is also welcome.

As for other models, many DNN models are based on decades of research and build on top of that.

An entirely novel architecture may not necessarily fare as well on its first outings.

[AMZ]

91% classification accuracy on MNIST benchmark is still not acceptable in machine learning community, but this is for an HTM network with 100 mini-columns. Increasing the number of mini-columns to 2048 can bring the accuracy up to ~ 95-96%.

[rhyolight]

As others have said, the accuracy is not the interesting part of the story. The interesting part is the neuromorphic architecture.