Well really I don’t know. A tally is a sum anyway. It also would be a bit demanding for biological neurons to implement switched linear projections. Whatever biology implements, it has to be very robust.
Anyway some sort of conversation about matters went on here:https://github.com/max-andr/relu_networks_overconfident/issues/2
To which I would add or did add:
"Only 1 non-linearity per N weights in conventional artificial neural networks !!!
In a conventional artificial neural networks there are n weighed sums operating on a single common vector of non-linear terms (the output of the previous layer.)
One obvious problem is it takes n^2 operations to process the n weighted sums. Ie. A lot.
A less obvious problem is there are n weight parameters for each non-linear term.
The greater the number of non-linear terms the greater the ability to separate inputs into different decision regions and the non-linear terms also reduce correlations.
In fact, if the weight vectors in each weighted sum in a layer in a conventional neural network are not orthogonal there will be correlations between the outputs of the weighted sums.
Using varied random projections of a common vector followed by application of non-linear functions it is possible to give each weight parameter its own non-linear term. There is one weight parameter per non-linear term.
That got me thinking about the weight efficiency of ReLU versus switch slope at zero (f(x)=a.x x>=0, f(x)=b.x x<0) in fixed filter bank neural networks or similar.
I think ReLU would win out on the efficiency basis because you are getting more Independence per weight. However you lose the possibility of getting free ResNet like transport of information to where it is needed in the network that is possible with switched slope.
The system, while designing itself can decide to pass information straight through (by setting a=b=1).
With ReLU an axe must sometimes fall and information must fail to get through about half the time. ReLU would also likely work a mischief with the behavior of the filter bank.