Even in On Intelligence, Jeff thought the classic view of hierarchy was correct, but after learning about grid cells and realizing how they might be used in the neocortex to model objects in allocentric spaces, we had to rethink the hierarchy to make it work. HTM theory posits that a location signal (via grid cells) exists for each cortical column representing some unique point in space. This assumption broke our classic idea of hierarchy if we continue to apply the Mountecastle model of a cortical column.
If we assume every cortical column is representing some feature at some location in an object’s space, this just doesn’t make sense in the classical view of hierarchy, because each region of the classical hierarchy is identifying different objects and composing them. We don’t think it works like that anymore. This new model of hierarchy explains a lot of things, including object composition by association. stay tuned for podcast
If there was a black box in front of you and I asked you to reach in with your hand and tell me what it was quickly, it would take you about 0.5 seconds to grab the coffee cup, recognize across your hand hierarchy at all levels that it matches coffee cup, and tell me right away.
However, if I asked you to reach in with your foot and tell me what object was inside, it would take you much longer, and you would probably find yourself touching and thinking “what was that feature, a rim? oh, that’s a cup then”. That’s because your foot hierarchy has never learned coffee cups. It knows only the world of feet, like socks and shoes, lint, sand, coins, and the occasional unfortunate wad of gum. That part of your brain has no models of coffee cups unless you teach it (like every other part of your brain).
So how did you recognize it with effort? Think about the hand and foot hierarchies as different structures that eventually converge together at the high levels. Your foot has no idea what a coffee cup feels like, so it passes simple features like rims and curved planes and smoothness up the hierarchy until it can match it across columns that link downward in their hierarchies to different sensory areas!
That is the 2nd essential point here, that cortical columns train each other as they learn. That’s how we think this learning transfer happens at all levels of the hierarchy. Just like this, at lower levels of the hand hierarchy your fingers train each other as they learn things. That is how you can touch something with one finger to learn it and recognize it with another finger, even on another hand. This lateral learning must be happening across cortical columns at all layers of the hierarchy.
To be clear, we don’t know exactly how this works yet. But we have thought about it a lot and are working on solving it.