In case you haven’t seen it this cortical io video gives a good and brief summary of their system. It uses a corpus as well to form the semantic map, and takes advantage of the power of SDR’s to capture a lot of meaning in a small space, even whole sentences at once. I’m no expert at all on NLP, just a big fan of the htm-based approach because our brains have to capture a lot of meaning and they have to do it efficiently without a lot of math and energy use - a constraint that DNN’s don’t seem to generally adhere to.
1 Like
I hate to rock the boat here but the formation of the Cortical IO retina involves the use of those nasty old traditional techniques that that @maxlee was just outlining.
In this case, they use a SOM technique.
3 Likes
Yes they certainly depart from neuroscience in forming the semantic map as you’d know much better than I. Ultimately this map should be formed in an online fashion the way the rest of HTM theory does, a fascinating question how that happens. In the absence of that understanding they’ve used some non-biological machinery to fill the gaps.
I know SOM isn’t the only such machinery they use, though it seems more similar to HTM than any other ANN mechanics I know of (a little like learning in SP at least). Though if the SDR’s they produce are viable in their semantic overlap their system seems good as any for the time being (?).
1 Like
They use HTMs to read out the data after it is organized.
I am working though how to form the maps with online learning and it seems to be a very hard problem.
1 Like
I am working on the same problem. I’ve used eligibility traces effectively for the online part, but still working out the topology element. Current theory is that grids could be used for this, but haven’t gotten into the weeds with that idea yet.
1 Like
There are a large number of people that have no clue how to write but are perfectly fluent in a language.
I think that you can say that letters/writing is an artifact to capture either syllables (most alphabet based languages and Hangul) or word sounds (pictograph based languages).
By the same token, the stream of syllables to create the elaborate alert, mating, and dominance calls (the basis of language) that animals use for signaling is an artifact of our biological sound production hardware.
If you are looking to create biologically based systems and are trying to decide on the correct level to match up with the human neural hardware I would offer that our human semantic grounding in the speaking-hearing axis along the arcuate fasciculus pathway may be the best place to start. (Syllables)
Hangul: possibly the best match between spoken syllables and writing?
Hi Mark, Sorry it took a while but I had to figure out what I thought. You’re likely correct that Hangul is the best text to speech representation out there but it is only efficient in Korean and more importantly it is not particularly relevant to this problem. This problem being thinking. If I look at what I wrote and your response I think my use of the the word syllable sent us down a blind fork. What I might have said instead is phoneme and still that suggests speech. So I might have said thought chunk but what does that mean and how does it relate to a Chinese character generator or Hangul for that matter. This is where I had to stop and think about an explanation.
First I should say that in ’77 I was not trying to create or even approximate a biologically based system. I was trying to learn Chinese and wanted a database that contained all of the information required. That required a stroke by stroke visual representation of the characters along with the sound, including tone, the meaning, the part of speech and more. The sound was just a part. The thing I was most pleased by initially and what I presented to Apple was a hexadecimal way of writing the character. The way it works is just a reimagined hierarchical stack of plastic ascii. The first level is stroke, the second is radical ( of which there are only 214 ) and the third level is character. It was not until I considered the forth level of idiomatic expression or compound word that I began to think of this as a thought processor rather than just a word processor.
It turns out that this system can draw Korean characters as well as Chinese but it also can write any other language in the form letter, syllable, word. This is why it doesn’t matter that Hangul is more efficient than English at describing the phonetics of a language. it is not the sound of the language but the meaning. In this respect it is important to see the syllable as a thought chunk, or a Latinate, Indo-European or older root. One of the interesting aspects of Chinese is that each of the 214 radicals has several thousand years of evolution and carries rich pictorial and metaphorical information in addition to the aural.
At any rate all of this is to say that language is not thought. As much as we might perceive of our thought in terms of language is is consequence and not a cause. For this reason any method of describing the sound is equally good because none of them are what is going on at the fundamental level of thinking. One thing we seem to have discovered is that thinking sounds a lot like a rather binary static if we overlook the meaning of the amplitude of the spikes that is. And as we know, that amplitude is anything but trivial. But the spoken or written or danced language those spikes and intervals ultimately get expressed in is trivial.
2 Likes
Thank you for your thoughtful reply.
I agree - syllables are a stand-in for phonemes. For the bystander watching this exchange:
When I was working with the AVOS company (Assistive technology for the visually impaired) I became very interested in speech IO and the production of speech sounds. I discovered that the sounds of speech are an artifact of the kinds of sounds that the human speech production hardware was capable of making.
Please examine the charts on page 15 of this lecture:
http://research.cs.tamu.edu/prism/lectures/sp/l3.pdf
You can see that there is are maps of the various sounds that a human is capable of making and the sounds are “fixed points” on these maps that are distinct enough that they can be reliably produced and recognized.
I totally agree - speech sounds are not thinking. I suspect that this is a large part of the reason that AI approaches that are text/speech based have not been very successful. That said - speaking really does engage and expand the mental hardware. Humans without speech really are not what most of us think of a fully human. I have spent a fair amount of time thinking about this and have commented on this before:
While it sounds difficult to extract and represent the underlying symbols of the communication of thought it seems that google is mucking around with that very thing with their translate project:
Again - for the interested bystander - Want to learn more?
This is one of the first books I read on these topics, A classic top to bottom text on the entire chain of speech from speaker to listener:
3 Likes
I travel extensively for my job and have been doing so since the early 1990s. This included frequent visits to China and it became useful to learn Mandarin Chinese. I used the Pimsleur course and eventually was able to function on my own in day-to-day interactions. Immersion in the language and culture really drives this home. I can read and write a bit but my tones are terrible. I understand what you are saying about Chinese stroke order and character formation. I am struck that in Chinese many of the pictograms are actually reasonably good renditions of the thing that they are communicating.
Some examples:
火 - fire; I love this one = stick-figure person running around FIRE!
水 - water; hard to see the original picture - see chart below.
山 - mountain
口 - mouth/opening
品 - goods/commodities; a pile of boxes
門 - door; western movie bar doors anyone?
In some cases, the original picture has evolved to be difficult to recognize - for example - water (shuǐ), fourth line in this chart.
(evolution/versions of Sun, moon, mountain, water, rain, wood)
This is an ongoing process. For example, the door has evolved from the swinging bar doors 門 of the traditional Chinese to the more generic 门 in simplified Chinese.
The combination of these symbols is also somewhat based on more than just strokes or radicals; they often tell a short story.
日 - Sun (rì) - also used to say “day” things, with 月 - moon (yuè) to say “month” things.
間 - Time; you look to the door to see the sun position, learning the time.
Putting things in a door is used for other constructions
心 - heart
悶 - Stifling - you are inside but your heart is out the door.
But marking an actual door?
出口 - exit; mouth/opening to the mountains/outdoors.
Chinese is filled with these short stories; I love this language. There can be layers of meanings in a line.
What is not found in stroke order, (or simple letter sequences or syllables for a western script) is any sort of useful semantic or grammar information. This information is loaded at the word/pictograph and symbol grouping level. Any sort of generation algorithm will have to consider this level if the output is to look like it is making any grammatical sense. Languages that use conjugation have to consider word grouping to influence construction at the word level. There is no local information at the word level that tells me if the word (tener in Spanish - “to have” or “to be”) I am building should be tener or tengo or tienes or tiene or tenemos or tenéis or tienen or (there are a bunch more). This strongly suggests that the higher levels will feedback to the lower levels in word construction.
But this will still be gibberish. Without any semantic guidance, the generation from a system that follows reasonable grammar rules will make something that looks like the human affliction Wernicke’s aphasia.
Note the huge range of signs and symptoms listed. This gives some indication of the range of factors that go into speech production. The clusters of defects suggest to me that the production of words is the product of several maps working in harmony, each contributing to some aspect of the ongoing production stream.
This tells us that if the map or connections to it are failing you suffer the related defect. From a modeling perspective this suggests your functional building blocks.
A little more on structuring these building blocks; a key difference between biological hardware and computers is that computers have variables, brains have connections.
- An important programming task is to learn the producer(s) and consumer(s) of a chunk of information and WHEN and WHERE it is produced and consumed. If a chunk of information in a computer is needed in several places it is tucked into a storage space and accessed wherever it is needed.
- In the brain information exists in SOME FORM in SOME PLACE in the brain. If there is a producer and a consumer there has to be a connection. If there is some order or stages to this process there has to be a physical pipeline. Parts of these pipelines may be selectively enabled to gate the flow of information but the connections are always there. If you think about it, there is no other way that neural hardware can work.
Whenever I see someone that is describing some AI proposal I keep this distinction firmly in mind to test if it biologically plausible. Once I started thinking this way it shaped how I view papers describing neural research and proposed architectures.
2 Likes