SWNet16 neural network: https://archive.org/details/sw-net-16-b
Fuses multiple width 16 CReLU layers into one larger layer using the one-to-all connectivity of a fast transform.
Then stacks those layers into a neural network.
Does spectral de-biasing at the input and output using permuted Thue-Morse sequence sign flips.
Java source code.
I did a version in Processing (Java generative art):
https://discourse.processing.org/t/swnet16-neural-network/47779
Which is a single file to look at.
There is some evidence that artificial neural networks are only a form of hierarchical memory.
There is evidence for feature extraction and I find that evolution training finds no improvement after gradient training. Also no algorithms as such found to emerge, for example any bubble sort type algorithm, or other simple algorithm.
They are all consistent with hierarchical memory though. Features allow selective memory, evolution cannot improve already fully learned memory, though evolution could adjust an actual algorithm to at least find some small improvement.
That kind of leaves a mystery of how artificial neural networks can generalize in smart ways.
Some recent papers show that small networks trained with modular arithmetic problems learn to generalize by the formation of completed geometric forms inside the net. When test time data is inputted into the net it lands on the geometric manifold and elicits the correct response as a consequence.
If you say an artificial neural networks is hierarchical memory then factorized geometric forms can exist in the network, allowing even more complex generalization.
So first you are accusing artificial neural networks of being much simpler than everyone is saying. Then you are pointing out despite that simplicity, complex emergent factorized geometric forms are learned and allow reasonable forms of generalization to happen.
I put together a comic book outline:
https://archive.org/details/fast-transforms-for-neural-networks
Maybe it will get a bit of traction, like the Britney Spears’ Guide to
Semiconductor Physics.