"How Your Brain Organizes Information" video

[robf]

Let’s use the word “factorize” as you use it:

@CollinsEM

Language itself can be factorized on multiple levels and from multiple perspectives (characters, tokens, phonemes, words, phrases, sentences, grammar, semantics, etc.). Scholars have picked apart language from almost every conceivable angle.

@CollinsEM

When I say factorize, I’m using it in the sense that any spatial and/or temporal signal can be decomposed into coherent and incoherent components.

…
The coherent composent … is any combination of signals that can be represented via a mathematical expression.

OK. So we’ve got “Language … can be decomposed into … a mathematical expression.”

What is that mathematical expression?

I say it can’t.

I say language (“natural” human language, to distinguish it from computer or artificial languages) cannot be completely described any more compactly than a full body of its usage (that’s what distinguishes natural language from computer language or mathematical languages, and what has prevented us from building effective AI models of it up to now.)

You may say that mathematical expression exists, it has just not been found yet.

@roboto said he too was “sure” it was possible:

@roboto

In my opinion there’s a way to somehow describe the structure of language in a statistical manner and it’s just that maybe we haven’t found it yet.

Then @DanML posted a book supporting something, but resolving to me, as supportive of what I was saying:

@robf

Exactly what idea are you saying is supported by Christiansen/Chater, @DanML?

In the Penguin summary of The Language Game you link to they say:

“Upending centuries of scholarship (including, most recently, Chomsky and Pinker) The Language Game shows how people learn to talk not by acquiring fixed meanings and rules, but by picking up, reusing, and recombining countless linguistic fragments in novel ways.”

Over and over in this thread I’m repeating, with evidence (actual contradictions in language structure, and the size of language models, to name two) that natural language structure cannot be (completely and consistently) abstracted. But people just keep repeating that they are “sure” it is possible. It’s kind of amusing. But repetition as a public service also gets tiring.

To avoid repeating myself you should direct your arguments to what I wrote e.g. in this thread. Firstly evidence from the size of transformer models:

@robf

The question was, as @roboto hypothesized, whether the parameters represent any kind of simplification of the data.

Or, as I hypothesize, the parameters actually represent a kind of expansion of latent structure in the data, and there’s no limit to how much increasing them might improve the model.

Maybe this post, #4 in this thread, gives the most comprehensive collection of arguments I’ve presented here on it:

"How Your Brain Organizes Information" video

Exactly the problem I’ve been talking about with all my posts. The belief there will be factors is hard to get past. We see factors. Objects. That’s all we see. It is only when you try to nail them down computationally that things start to come apart. I believe the inconsistency of the classes/“edges” you find is what leads to parameter blow out with large language models. That’s the reason they’re “large”. And, together with the fact that they hide grammar, why they have done so much better …