Given a context at each layer that selects parameters for a linear mapping, training produces the best average linear mapping for that context.
Then the entire networks is a composition of best average linear mappings.
https://archive.org/details/a-context-based-view-of-deep-neural-networks
You can take the:
Red Pill (pdf)
or the:
Blue Pill (Blog)
They are both the same.
2 Likes
If you want to take the:
Orange Pill
It includes a brief discussion of the decision matrix view of ReLU. You can also view the decision matrix as a binary context matrix.
In such system there can be a lot of weight sharing going on (different contexts share some parameters.)
So you need to speak of ‘best efforts’ in terms of linear mapping learning.
If contexts are only a small Hamming distance from each other then they can even benefit from parameter sharing.
Overall the described systems are like decision trees with linear mapping river sections that connect up together and allow feature data to flow through and metamorphose through.
Quite interesting, who knew?
If you can accept a cartoon version:
https://sciencelimelight.blogspot.com/2026/07/dora-explorers-metamorphic-river.html