# I implememted a dense Spatial Pooling layer

**URL:** <https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272>\
**Category:** Lounge\
**Created:** [January 12, 2018, 3:03pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272 "2018-01-12T15:03:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [January 12, 2018, 3:03pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/1 "2018-01-12T15:03:46Z")

</div>

Hello all  
I’m new to HTM. I saw Siraj’s new video explaining how HTM works this morning. After finding some resources detailing how HTM works. It seems that HTM has trouble a dealing with dense data (HTM uses SDR).  
So I wrote a special fully-connected layer that reserves dense data as input (ordinary tensors used in deep learning) and trains(and operate) itself the same way a Spatial Pooling layer in HTM does. - By comparing inputs to it’s weights and updating weights by looking at the activation state of each input.

To test if my layer works. I connected my layer to a fully-connected layer to produce a classifier(the FC layer is trained using backprpoergation). I can get my classier to classify MNIST digits correctly around 40% after traning.

Is this idea of doing HTM using dense data worth anything? Do anyone has any idea on how could I improve my design?

[Here](https://gist.github.com/marty1885/a80138f518283eb5db0256acd78e1020) is the code. It is implemented using my own deep learning library. (available [here](https://github.com/marty1885/Athena))

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 3:22pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/2 "2018-01-12T15:22:54Z")

</div>

Hi Marty! Thanks for joining our community. You have some misconceptions about HTM. I’ll try to direct you to some resources that can help you understand better.

> [@marty1885](#):
>
> It seems that HTM has trouble a dealing with dense data

SDRs are very important. HTM is a theory of intelligence in the brain. SDRs represent neuronal connectivity, and this activity is sparse in the brain. The sparsity is important. For more info, see [this video](https://www.youtube.com/watch?v=ZDgCdWTuIzc).

> [@marty1885](#):
>
> So I wrote a special fully-connected layer that reserves dense data as input (ordinary tensors used in deep learning) and trains(and operate) itself the same way a Spatial Pooling layer in HTM does. - By comparing inputs to it’s weights and updating weights by looking at the activation state of each input.

We are operating in very high dimensional spaces. When representations are sparse, the are easier to compare with each other to derive similarity. Sparsity is a requirement of intelligence, as we understand it in the brain. Making it dense will break everything.

> [@marty1885](#):
>
> Is this idea of doing HTM using dense data worth anything?

Honestly, I don’t think it will work at all. I think density breaks HTM. But I’ve been wrong before. 😉

Anyway thanks for the work you put into this (although the gist you gave is broken).

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [January 12, 2018, 3:35pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/3 "2018-01-12T15:35:44Z")

</div>

I’ve watched all tutorials from HTM School.  
AFAIK neurons in our brains can communicate using real values by spiking in different frequency or delay in time. How is this captured in HTM?

I update the broken link.

Thanks for your reply.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 3:41pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/4 "2018-01-12T15:41:05Z")

</div>

> [@marty1885](#):
>
> I’ve watched all tutorials from HTM School.

Good, I hope they were helpful.

> [@marty1885](#):
>
> AFAIK neurons in our brains can communicate using real values by spiking in different frequency or delay in time. How is this captured in HTM?

This is not captured in current HTM implementations. Oscillations (spike timing) will probably need to be tackled to complete sensorimotor theory.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 3:46pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/5 "2018-01-12T15:46:55Z")

</div>

I looked at your code. What is the input? Images?

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [January 12, 2018, 3:49pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/6 "2018-01-12T15:49:31Z")

</div>

> [@rhyolight](#):
>
> Good, I hope they were helpful.

They are. Thanks for making the videos! Keep them comming!

> [@rhyolight](#):
>
> This is not captured in current HTM implementations. Oscillations (spike timing) will probably need to be tackled to complete sensorimotor theory.

Got it. Thanks!

> [@rhyolight](#):
>
> I looked at your code. What is the input? Images?

Yes they are.  
I use MINST because they are easy to load and they are somewhat dense (comparing to the SDRs used in the tutoruals).

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 3:50pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/7 "2018-01-12T15:50:22Z")

</div>

So you are encoding images into SDRs in order to feed them into the Spatial Pooler? If so that makes sense. You’ve just created an encoder. I think I understand what you’ve done, and a lot of other people have done the same with the SP. You have to understand the temporal aspect of HTM as well. Temporal data over time, and the ability to memorize sequences of complex spatial data in a way that is generalizable… that’s the power.

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [January 12, 2018, 4:09pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/8 "2018-01-12T16:09:38Z")

</div>

Hmm…  
I started out by trying to solve the problem of encoding complex data into SDRs. (as mentioned in ep. 6). So I decided I’ll implement a SP that accepts dense inputs so no encoder is needed. The SP will deal with how to encode the data by it self.  
I was planing to stack DSPs(Dense SPs) together and it should behave as normal SPs. And theoretically I can replace all layers in HTM with dense versions of them and it should work and also encode spike timing as a bonus.  
But currently I’m just making a PoC to proof that DSPs can work, And made my DSP output SDRs because I can’t get DSP to converge enough for the following FC layer to do anything useful.

Are similar methods used to encode images? I didn’t find anything when I’m searching on Google.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 4:20pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/9 "2018-01-12T16:20:36Z")

</div>

> [@marty1885](#):
>
> I’ll implement a SP that accepts dense inputs so no encoder is needed.

I think maybe you’re thinking of creating an [autoencoder](https://discourse.numenta.org/search?q=autoencoder), which has been talked about before here?

> [@marty1885](#):
>
> I was planing to stack DSPs(Dense SPs) together and it should behave as normal SPs.

So will a “Dense SP” output a dense representation? If so, no TM will work on it. Also I don’t understand what you mean by stacking. Data → encoding → SP → TM → classification. There is no stacking in this model until you start talking about [layers and columns](https://numenta.com/papers/a-theory-of-how-columns-in-the-neocortex-enable-learning-the-structure-of-the-world/).

> [@marty1885](#):
>
> But currently I’m just making a PoC to proof that DSPs can work, And made my DSP output SDRs because I can’t get DSP to converge enough for the following FC layer to do anything useful.

At this point I’m pretty sure you’re talking about some type of autoencoder that can convert any dense representation into a sparse one, before sending data to the SP. That’s not a “Dense SP”, it is just an encoder. Do you understand why I say that?

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [January 12, 2018, 5:28pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/10 "2018-01-12T17:28:05Z")

</div>

> [@rhyolight](#):
>
> At this point I’m pretty sure you’re talking about some type of autoencoder that can convert any dense representation into a sparse one, before sending data to the SP. That’s not a “Dense SP”, it is just an encoder. Do you understand why I say that?

Sure. Thanks.

I should have post after I got more stuff instead of what I have now. I’m working on the problem that my layer will collapse(output becomes 0) if I make it output dense data instead of SDR.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [January 12, 2018, 5:36pm UTC](https://discourse.numenta.org/t/i-implememted-a-dense-spatial-pooling-layer/3272/11 "2018-01-12T17:36:38Z")

</div>

Let us know if you find anything interesting. But keep in mind that if the SP outputs a dense array, HTM sequence memory algorithms will not work on it.