What is next?

[robf]

@Jose_Cueto

My first thought is that we don’t have yet an algorithm to update model weights with fine-grained values or that mimics gradients. I don’t know if the brain is computing these gradients though.

I agree. I don’t think the brain is computing gradients. Though I do think the solution is seeking predictive energy minima. So the same goal as computing gradients, just a different way of finding minima of those gradients.

Instead of computing gradients, I think the brain finds predictive energy minima as network resonances using oscillations.

No need to update model weights with fine-grained values. And with the additional benefit that the energy minima can vary dynamically. Perhaps explaining why LLMs have such enormous parameter blow outs.

I outlined what I think is the appropriate contrast here:

@robf

The “algorithm” is to find prediction energy… minima. In that sense it is the same as transformers/LLMs. The prediction energy minima LLMs seek, are those on energy surfaces formed by collections of elements, notably back along the sequence, with “attention”. They may also seek energy minima on energy surfaces formed by collections of elements representing the current state …

My “algorithm” is the same. It’s seeking minimum energy prediction collections across a network. The main difference is that I don’t think the minimum energy predictive state is static across the whole data set. I think there are different best minimum energy predictive states depending on what you are most interested in. In particular I believe that minimum energy predictive state will vary from prompt to prompt. (I think this is happening in transformers too. But we don’t see it. They hide it from us in their enormous context sensitive “parameter” sets.)

So I want to find the minimum energy predictive state dynamically. And a way to do that should be to set the network oscillating and allow it to find its own minimum energy states, as resonances on the network.