Some Aspects of Backpropagation

Community Lounge
1

Understanding backpropagation as a linear update with deferred non-linear effects:
https://sites.google.com/view/algorithmshortcuts/some-aspects-of-backpropagation

The thing I learned from writing about it is if you want to use locality sensitive hashing to select weights, weight vectors or weight matrices from a pool you better adopt some tactic to mitigate the over-response to minor Gaussian noise changes.
Possibly by some kind of blending technique.
I will go away and ruminate about it.

2

It seems like you should use locality sensitive hashing to select individual weights from a pool of weights, rather than complete weight vectors or matrices. That seems less likely to disrupt backpropagation with excessive non-linear behavior. Getting one weight wrong in a n dimensional weighted sum is not especially disruptive, for example.