# Step activation functions

**URL:** <https://discourse.numenta.org/t/step-activation-functions/2295>\
**Category:** Lounge\
**Created:** [May 7, 2017, 3:27pm UTC](https://discourse.numenta.org/t/step-activation-functions/2295 "2017-05-07T15:27:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [May 7, 2017, 3:27pm UTC](https://discourse.numenta.org/t/step-activation-functions/2295/1 "2017-05-07T15:27:04Z")

</div>

Recently I recognized more clearly that step activation functions in single layer neural networks give the best performance in terms of learning speed and separation of similar inputs.

I use the signof function:  
fn(x) = 1, x\>=0  
fn(x) =-1, x\<0

Or a soft version:  
fn(x) = sqr(x), x\>=0  
fn(x) =-sqr(-x), x\<0

There are very fast bit hack versions of the square root function if you need them.  
Anyway this paper provides some justification:  
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3921404/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3921404/)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [May 8, 2017, 12:03am UTC](https://discourse.numenta.org/t/step-activation-functions/2295/2 "2017-05-08T00:03:26Z")

</div>

There was a paper indicating 4 to 16 demultiplexing occurring in the brain. That is also a sort of step function (4 binary inputs light up one of 16 outputs.) If I do 1 to 2 demultiplexing in single layer nets I get great generalization ability but rather slow training. If I do 4 to 16 demultiplexing I get less good generalization but much faster training. Obviously there are trade offs, and perhaps evolution picked one for us?

---

<div class="post-metadata">

**Author:** ![Vinay6666](https://avatars.discourse-cdn.com/v4/letter/v/4da419/32.png) [@Vinay6666](https://discourse.numenta.org/u/Vinay6666)\
**Post date:** [October 17, 2017, 12:17pm UTC](https://discourse.numenta.org/t/step-activation-functions/2295/3 "2017-10-17T12:17:05Z")

</div>

> **[Activation Function | Quick KT](https://quickkt.com/tutorials/artificial-intelligence/deep-learning/activation-function/)**
>
> activation function, Sigmoid / Logistic, Softmax, Identity, Binary Step, ReLU

  
visit here for video tutorial on activation functions