# Why not apply SGD after every data sample?

**URL:** <https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453>\
**Category:** Machine Learning\
**Created:** [August 13, 2019, 3:51pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453 "2019-08-13T15:51:51Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [August 13, 2019, 3:51pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/1 "2019-08-13T15:51:52Z")

</div>

My first thread on the #engineering:machine-learning forum! So it is fitting that this is probably a complete n00b question. 😊

**Why not just apply Stochastic Gradient Descent (SGD) after each data sample instead of batching them into epochs?**

This would give the data a temporal context, and allow us to more easily code in Temporal Memory logic into hidden layers.

I assume the answer is because it is too computationally expensive? But that is not obvious to me.

---

<div class="post-metadata">

**Author:** ![mrcslws](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/mrcslws/32/4961_2.png) [@mrcslws](https://discourse.numenta.org/u/mrcslws)\
**Post date:** [August 13, 2019, 4:29pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/2 "2019-08-13T16:29:44Z")

</div>

See section **Mini-batch gradient descent** of [https://cs231n.github.io/optimization-1/](https://cs231n.github.io/optimization-1/)

Also note that epochs and batches are two different things. In each epoch we train the model on the entire training dataset, splitting it into some number of batches.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [August 13, 2019, 4:32pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/3 "2019-08-13T16:32:06Z")

</div>

Thanks, and another article I’m reading on this to inform the discussion (thanks @subutai) called [_A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size_](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/).

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [August 13, 2019, 4:34pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/4 "2019-08-13T16:34:08Z")

</div>

If you are talking about neural networks and training them on temporally-coherent data. No, this is a super, extra super bad idea. Unlike HTM, NNs are pron to catastrophic forgetting. Where after a few weight updates that doesn’t take account of the original function distribution, the NN completely forgets how to perform those functions. (they overfit to your small sample of data)

This is also why we shuffle the dataset before training.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [August 13, 2019, 5:07pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/5 "2019-08-13T17:07:40Z")

</div>

This was obviously an uninformed question, but I appreciate the responses. The material was very helpful. They were also helpful on Twitter:

> <https://twitter.com/rhyolight/status/1161308290866769920>

I think I understand the problem with SGD. The “stochastic” part of SGD basically disallows any type of fine-grained temporal context to get into the system. Applying GD in batches gets you part of the way there, but the mean in the cost function is necessary to remove noisy gradients, which would prevent the GD algorithm from finding a _decent_ **descent** 😉 .

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [August 13, 2019, 5:25pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/6 "2019-08-13T17:25:43Z")

</div>

… unless of course you are feeding time in as one of the axis parameters to be part of the manifold formed.

Not the sequence of presentation but as part of the data set.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [August 13, 2019, 5:29pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/7 "2019-08-13T17:29:24Z")

</div>

Do you know of any experiments where this was useful?

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [August 13, 2019, 5:33pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/8 "2019-08-13T17:33:05Z")

</div>

Some of what we work with is time sequences of response to a tire passing over the scale.  
There are many factors involved in both the wheel/chassis system of the truck and the various spring-rate and masses of the scale assembly.

I have experimented with this and so far - have not achieved enlightenment.

The hope is that I will find an application of a neural network at work, in this case, the filter section. Alas - not yet.

---

<div class="post-metadata">

**Author:** ![hsgo](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@hsgo](https://discourse.numenta.org/u/hsgo)\
**Post date:** [August 14, 2019, 12:48pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/9 "2019-08-14T12:48:08Z")

</div>

It’s also related to how GPU works.

Both the feedforward and the feedback happens in parallel per minibatch.  
Thus, it HAS access to all of the information of the previous “time step”(i.e. the actviations cause by the previous minibatch input).  
You can make use of the information if you want.

Individual backpropagations(gradient descent through the layers) aren’t computationally expensive either.  
They have about the same computational complexity as the feedforward stage has.  
The formula between the feedforward and the feedback stage are quite similar, too.  
That’s where it got its name, “backpropagation”.  
Training of deep learning models is costly because of the number of iterations it demands to work well.

What you’re suggesting(minibatch size of 1) might not be parallel enough to computed efficiently by GPU.

I’m currently implementing Deep HTM for GPU and indeed it has minibatch size of dozens.  
I’m not there yet, but I think it wouldn’t have any particular problem because of the fact that it does.  
If you think it might cause a problem of some kind, please let me know how.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [August 14, 2019, 2:16pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/10 "2019-08-14T14:16:51Z")

</div>

> [@hsgo](#):
>
> If you think it might cause a problem of some kind, please let me know how.

It would be extremely dependent on the type of data being processed and the semantic characteristics of the encoding.

---

<div class="post-metadata">

**Author:** ![hsgo](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@hsgo](https://discourse.numenta.org/u/hsgo)\
**Post date:** [August 14, 2019, 2:27pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/11 "2019-08-14T14:27:15Z")

</div>

I think that could not be the case because of the encoder layers that process the input and feed it to the SP layer.  
The input doesn’t get fed directly. Thus, it gets less affected by the type of the input.  
But there will be some kind of effect for sure.  
I’ll keep that in mind. Thanks! 🙂

---

<div class="post-metadata">

**Author:** ![michaelklachko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/michaelklachko/32/14_2.png) [@michaelklachko](https://discourse.numenta.org/u/michaelklachko)\
**Post date:** [August 16, 2019, 7:58pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/12 "2019-08-16T19:58:48Z")

</div>

@rhyolight, just to clarify SGD will work just fine with batch size of 1, but you will need to do a lot more weight updates. I trained small convnets like that, and saw no accuracy drop.

Ilya Sutskever [seems to agree](http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html):

> It would actually be nice to use minibatches of size 1, and they would probably result in improved performance and lower overfitting; but the benefit of doing so is outweighed the massive computational gains provided by minibatches.

In case you don’t know who Ilya Sutskever is: [Ilya Sutskever - Wikipedia](https://en.wikipedia.org/wiki/Ilya_Sutskever)

---

<div class="post-metadata">

**Author:** ![michaelklachko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/michaelklachko/32/14_2.png) [@michaelklachko](https://discourse.numenta.org/u/michaelklachko)\
**Post date:** [August 16, 2019, 8:03pm UTC](https://discourse.numenta.org/t/why-not-apply-sgd-after-every-data-sample/6453/13 "2019-08-16T20:03:35Z")

</div>

Oh, and for temporal context you would typically use attention mechanisms. It’s strange no one has mentioned that.