Esperanto NLP using HTM and my findings

I have avoided this “supervised learning” issue on this forum as is seem to be close to a “religious or political” third rail.

When it comes to language there is no example I can point to where a system forms internal models of functional language by “just listening.” There is always a teacher.

In the dad’s song project we determined that the “sexy dad” bird sang a song that was learned as the guiding training signal. Baby bird learned to reproduce the whole song as an entity. In this model, there was no underlying model of semantics. Entire songs are the words learned. This does not help us much with language but it could be very helpful in getting from letters to words.

In the human model, we have an external teacher to pair word sounds to perceived objects and sequences. These sequences could be verbs and relationships. If you expect the model to form internal models I think it is perfectly biologically plausible to have some sort of training signal paired with the stimulus to form this internal model.

How should we train up a hierarchy to form these internal models? Most self-organizing models I have seen have been underwhelming; the signals that form a structure in the presented data take massive repetition to discover these underlying patterns and are usually brittle. What is learned is often some irrelevant features that miss the “essential” structure of the data.

Not all.

In the “three visual streams” paper they have a bi-directional hierarchy with some primitive training on one end of the stream and the inputs on the other end. The “lower” end is the senses. In the paper, the “top” end is some facts about the physical world (object permanence, connectedness, and such) that provides feedback as the "The weak guidance on the “top” end and serves as a “light at the end of the tunnel” to help form internal models. For language, this guidance could be facts about the language such as semantic structure or word pairs. Even if the entire rest of the paper is utterly useless to you this principle is extremely valuable.

I will add that Numenta is just getting to the point where they are looking at the thalamus streams as part of the HTM canon. Work in this area could well be useful in advancing the HTM model.

1 Like