Fast Transform (aka. Fixed Filter Bank) neural networks trained by evolution and by backpropagation.
A fast transform (Walsh Hadamard Transform) is used as a fully connected set of fixed weights which, if used a conventional neural network activation function would give you a real but completely nonadjustable neural network layer.
Something must bend. What you can do is use individually adjustable parametric activation functions.
With switching based parametric activation functions the resulting fast transform neural network is very similar to a ReLU based network, though faster to compute and using fewer parameters per layer.