If you want to understand the basics of the Walsh Hadamard transform and some useful applications for machine learning you can look here:
https://www.kdnuggets.com/2021/07/wht-simpler-fast-fourier-transform-fft.html
I actually have a second article in with kdnuggets called “The Weighted Sum a Second Glance.”
Maybe they will accept, maybe they will reject.
It is really the math to construct general associative memory using the weighted sum. However, I did not say that directly, out of politeness.
There was active research into associative memory in the 1960s, even using very advanced hardware for the time:
https://archive.org/details/DTIC_AD0641205
1 Like
Fast Transforms for Sparsity:
https://discuss.neuralmagic.com/t/fast-transforms-for-sparsity/41
It is also interesting that a single weighted sum layer in a conventional dense neural network can embed the Walsh Hadamard transform and presumably many other simple transforms. For example the 4-point WHT of a,b,c,d is just
a + b + c + d
a - b + c - d
a - b - c + d
a - b + c - d
Which obviously you can construct with weighted sums using 16 fused multiply-adds compared to 8 add subtracts with the fast WHT. You can then think about issues like meaningful zero-curvature initialization of neural networks. Though you would have to use parametric activation functions.
ReLU forces extreme initial curvatures on any network which may ultimately lead to its replacement. It has been very successful anyway.