When I look at what deep neural networks do

[Bitking]

@Sean_O_Connor

Also it seems to me there are problems with lots of unwanted boundaries that are hard to get rid of.

Associative memory alone cannot replace what deep networks do in terms of conditional decision regions. Where if a vector is in one decision region in a given layer it will lead to a conditional choice between some attractor states in the next layer.

This made me think about the works of William H. Calvin. He did some pioneering work on a hexagonal organization of neural information and described some of the exact situations you describe. If you note the similarity to the current work on grids his work becomes very relevant to the current discussion.

COMPRESSING THE CEREBRAL CODE - William H. Calvin
http://williamcalvin.com/socns94.html
http://williamcalvin.com/bk9/bk9ch6.htm
The entire book:

THE CEREBRAL CODE by William H. Calvin (MIT Press 1996)