Marcus proposes that connectionist networks can only represent some subset of the training set used to build the network. This position simplifies to “I can only recall a remembered state.”
In a post response to the Numenta December 4 research meeting I outline a general method of building an object representation based on a collection of cortical maps or regions.
As this was already a very concentrated blast of material that would take perhaps 10 times as much supporting background material to provide a digestible path to take in the concept I stopped with what I had posted and hoped that someone would pick up on the concepts if there was any interest.
Alas - nada.
Reading the Marcus position drives home that this high-level view of cortical representation is not a mainstream view and that without this - the constructionist model is as impoverished as Marcus claims.
He does not think big enough. I see multiple object representation as I have outlined in a rough sketch - in particular - two in common use in the brain. The objects are composed of short fiber tracts, and the resulting objects are joined by the long fiber tracts.
The pair of high-dimensional representations most important to human speech are on either end of the Arcuate fasciculus, which joins the high-dimensional grammar (Broca’s area) to the core of high-dimensional object representation. (Wernicke’s Area) This allow a modular construction of object particles to populate the structure of language templates.
When this is combined with the serial stream of consciousness as described in this post, you have a speech production system that is vastly more complicated than most recurrent networks but at it’s core, works on the same basic concept.
As you may have noticed, you may not know how a sentence will end when you start it. You have some object(s) and relationship(s) you want to represent and you fire up some sentence form to encapsulate that object. Speech production is initiated and you perceive the speech as it produced. As the sentence rolls on the relationship part of the object lurking in the object store in the temporal lobe is selected and this goes back through the Arcuate fasciculus to prime the next part of the sentence production. This serial process is interactive between these two object stores, (grammar and object/relationship) each influencing the other to work cooperatively to form the sentence.
I have just described the production of external speech but in the process that we call thinking this process is retained internally and allows modular manipulation of the stored object fragments to form novel object relationships between stored object fragments. You can perceive this internal speech as an experience and both store and recall this as if it was perceived from an external source.
An important feature of this system is the novel recombination of sub-features and related generalization.
This is not the symbolic relationship that Marcus describes. It is the functioning of a properly configured connectionist system.
Of the prereading provided, this paper from Bengio comes the closest to what I am proposing here:
The Consciousness Prior - Yoshua Bengio