# Sparse Networks from Scratch: Faster Training without Losing Performance

**URL:** <https://discourse.numenta.org/t/sparse-networks-from-scratch-faster-training-without-losing-performance/8907>\
**Category:** Lounge\
**Created:** [August 31, 2021, 2:50pm UTC](https://discourse.numenta.org/t/sparse-networks-from-scratch-faster-training-without-losing-performance/8907 "2021-08-31T14:50:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mraptor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/mraptor/32/4469_2.png) [@mraptor](https://discourse.numenta.org/u/mraptor)\
**Post date:** [August 31, 2021, 2:50pm UTC](https://discourse.numenta.org/t/sparse-networks-from-scratch-faster-training-without-losing-performance/8907/1 "2021-08-31T14:50:52Z")

</div>

> **[Sparse Networks from Scratch: Faster Training without Losing Performance](https://timdettmers.com/2019/07/11/sparse-networks-from-scratch/)**
>
> This blog post explains the sparse momentum algorithm and how it enables the fast training of sparse networks to dense performance levels — sparse learning.

---

<div class="post-metadata">

**Author:** ![SeanOConnor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/seanoconnor/32/3441_2.png) [@SeanOConnor](https://discourse.numenta.org/u/SeanOConnor)\
**Post date:** [September 1, 2021, 11:49pm UTC](https://discourse.numenta.org/t/sparse-networks-from-scratch-faster-training-without-losing-performance/8907/2 "2021-09-01T23:49:18Z")

</div>

I like the overview of sparse neural networks here:  
[https://youtu.be/H7-p3OWPpEI](https://youtu.be/H7-p3OWPpEI)

And this video that is trying to tell you something:  
[https://youtu.be/QEWe-aRBUAs](https://youtu.be/QEWe-aRBUAs)

Anyway a single neuron in a ReLU layer forward connects to n weights in the next layer. In the positive activation state (x\>=0) ReLU is f(x)=x and the vector pattern defined by the n forward weights is projected with intensity x onto the next layer. In the negative activation state ReLU is f(x)=0, and nothing is projected onto the next layer.  
That doesn’t seem great to me. Perhaps you should keep the positive activation state behavior and change the negative activation state behavior. Have an alternative set of forward connected weights for the next layer and project _that_ vector pattern onto the next layer with intensity x when x is negative. That x is negative doesn’t really matter. Then the activation function is always f(x)=x, with the caveat that the activation function switches between 2 sets of forward connected weights in the next layer depending on x\>=0 or not.