Signal Propagation: A Framework for Learning and Inference In a Forward Pass
Adam Kohan, Edward A. Rietman, Hava T. Siegelmann
“Abstract—We propose a new learning framework, signal prop-
agation (sigprop), for propagating a learning signal and updating
neural network parameters via a forward pass, as an alternative
to backpropagation. In sigprop, there is only the forward path
for inference and learning. So, there are no structural or
computational constraints necessary for learning to take place,
beyond the inference model itself, such as feedback connectivity,
weight transport, or a backward pass, which exist under back-
propagation based approaches. That is, sigprop enables global
supervised learning with only a forward path. This is ideal for
parallel training of layers or modules. In biology, this explains
how neurons without feedback connections can still receive a
global learning signal. In hardware, this provides an approach
for global supervised learning without backward connectivity.
Sigprop by construction has compatibility with models of learning
in the brain and in hardware than backpropagation, including
alternative approaches relaxing learning constraints. We also
demonstrate that sigprop is more efficient in time and memory
than they are. To further explain the behavior of sigprop, we
provide evidence that sigprop provides useful learning signals
in context to backpropagation. To further support relevance
to biological and hardware learning, we use sigprop to train
continuous time neural networks with Hebbian updates, and train
spiking neural networks with only the voltage or with biologically
and hardware compatible surrogate functions.”
A fatal blow to backpropagation?