And there they stopped

ReLU is a switch. It is either pass through or off.
The only thing it can switch in simple networks are dot products (weighted sums) because that is all there is left.
Then you can add in other dot product and switching mechanisms such as convolution and pooling.
And there for some reason (or no reason at all) they stopped.

In fact you can throw any dot product mechanism into a neural network, including the FFT and Walsh Hadamard transform. If you get stuck for something to parameterize, you can parameterize the activation functions.

I guess Numenta did some other things with dot products, but generally speaking there is an illusory barrier that actually you can walk right through.

Maybe they got stopped by a ReLU?

1 Like