If you think a context dependent chain of linear mappings is a good approach then you have a lot of opportunity to include context signals often seen in biological systems:
Principle: Separate Data from Control
The data pathway carries the quantities to be transformed. The context control pathway carries persistent state that selects, gates, or modulates the computation performed on that data.
Rather than encoding all contextual information within the feature representation, context is communicated through a dedicated control pathway that determines which local computation is executed. This separates what is being computed (the data) from how it is computed (the context).
Examples of context include behavioral state, temporal state, internal clocks, spatial or navigational state, physiological state, safety state, and task or mission state. These variables evolve on their own time scales and need not be rediscovered from instantaneous sensory input.
- Finite automata → behavioral context.
- Internal clocks → temporal context.
- SLAM or place cells → spatial context.
- Energy level → physiological context.
- Atlas/LSH bits → geometric computational context.
- User goals → task context.
There is a way to figure out how important a particular context type is:
https://archive.org/details/context-specialization-metric
And that is to look at the learned linear mappings after training. If the context is of zero use all the linear mappings will be the same (or as close as the system will allow.) Otherwise they will be differentiated.
A simple robotics viewpoint:
https://archive.org/details/a-simple-view-of-context-based-robot-intelligence
Meh-Compression in neural networks:
https://archive.org/details/backpropagation-a-context-selected-matrix-teacher
The data/control split makes sense in theory, but I’m curious whether the overhead ever tips the other way. Like if your context is relatively static or low-dimensional, does separating the pathways actually gain you enough to justify the extra machinery? Feel like a lot of the examples would naturally compress that boundary anyway.
1 Like
Yes, exactly. And suddenly you have a metric to measure the usefulness of a context.
A zero information context such as an entirely random one will result in backpropagation making all the linear mappings basically the same. Then you can look back after training and say, that layer is useless or that context is useless. And then redesign your neural network.
Or you could find that some kind of unexpected context was very important like a context based on the local weather (temperature, rainfall, windspeed.)
I don’t know if you can experiment with this kind of thing with the conventional neural network libraries.
This is a repeated problem.
I created fast transform based neural networks before and I would say no one else ever tried them because the ML/NN libraries don’t provide correct access (or any access) to fast transforms.
My code would all be in Java, JS, FreeBasic, Lua etc.
That’s not a me problem though. It’s a problem for other people.
There has been work done in switched linear systems by control theory researchers.
Their interests focus on system stability under all circumstances. Nonetheless switched linear control has been applied to robotics and so on. So it is not entirely new. Perhaps the connection to neural networks is new though, and training switched linear systems using backpropagation has not been looked at specifically in those terms (inadvertently though through ReLU neural networks.)
The new concepts open up a very large design space for neural networks. The context can be external to the data like robot speed, or internal to the data like geometry sensing of the input data with locality sensitive hashing, or the layer by layer determined context seen with ReLU neural networks.
It opens up the door to context engineering and a very wide variety of context addressed linear mapping choices.
I think it’s fun anyway.
I have said more or less all I can about it at the moment on Archive (click on uploaded by for everything.)
It’s time for me to code a lot of examples and experiments. I’m trying get away from doom-scrolling and get my head back into coding.
You are being asked to make a major shift in how you understand neural networks and ReLU neural networks in particular. Perhaps this can help: