# Fast FF networks speedups

**URL:** <https://discourse.numenta.org/t/fast-ff-networks-speedups/11124>\
**Category:** Machine Learning\
**Created:** [November 22, 2023, 11:57am UTC](https://discourse.numenta.org/t/fast-ff-networks-speedups/11124 "2023-11-22T11:57:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [November 22, 2023, 11:57am UTC](https://discourse.numenta.org/t/fast-ff-networks-speedups/11124/1 "2023-11-22T11:57:05Z")

</div>

Paper one [Exponential Faster Language Modelling](https://arxiv.org/abs/2311.10770) about using FFF in transformer FF blocks for training & inference.

Paper two [Fast Feedforward Networks](https://arxiv.org/abs/2308.14711) presenting the actual algorithm

It is a form of sparse execution (e.g. 0.3% of neurons being activated) instead of sparse representation (the HTM’s key feature).

Also interesting - “hacking” GPU for conditional matrix multiplication was used, and there is potential for native support in future hardware.

---

<div class="post-metadata">

**Author:** ![BrainVx](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@BrainVx](https://discourse.numenta.org/u/BrainVx)\
**Post date:** [November 22, 2023, 5:35pm UTC](https://discourse.numenta.org/t/fast-ff-networks-speedups/11124/2 "2023-11-22T17:35:56Z")

</div>

Quite interesting the inference pass works well with only 0.3% activation, it then raises the question as to how the learning phase could achieve a much lower activation during the learning pass.

Curious that RTE performance is significanltly worse and higher variability (1% activation 53.8 vs 0.5% activation 59.9 whilst 1.8% is 56.2) compared to BERT base. Could this indicate that those 90+% for RTE contain a high degree of subtle aggregate influences for inferred determination ?

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [November 22, 2023, 6:04pm UTC](https://discourse.numenta.org/t/fast-ff-networks-speedups/11124/3 "2023-11-22T18:04:48Z")

</div>

> [@BrainVx](#):
>
> Quite interesting the inference pass works well with only 0.3% activation, it then raises the question as to how the learning phase could achieve a much lower activation during the learning pass.

You mean to accelerate training too? That would be a more difficult problem, since training is done in batches and routing here (picking what neurons to activate) is computed from input on each FF block.  
The only way I can think of is training it with a batch size of 1, but that isn’t going to accelerate things too much. In which case it might not make much sense to even use a GPU.  
However,

- the gating algorithm might stabilize the slower converging stochastic gradient descent.
- parallel training on CPU would benefit from that each training step only updates 1% of the weights.