# How HTM is supposed to deal with spatial invariance?

**URL:** <https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051>\
**Category:** Numenta Theory\
**Created:** [November 11, 2017, 10:32am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051 "2017-11-11T10:32:32Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [January 30, 2018, 10:13pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/21 "2018-01-30T22:13:01Z")

</div>

Thank you for providing your vision of the whole picture with supporting links.  
Could you also elaborate on this statement?

> [@Bitking](#):
>
> We even know the native data structure format - distributed grids

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [January 30, 2018, 10:15pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/22 "2018-01-30T22:15:55Z")

</div>

> [@sebjwallace](#):
>
> convolution is an easy work around, but the brain doesn’t do that

I support this point of view, but could point to any supporting evidence?

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [January 30, 2018, 11:30pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/23 "2018-01-30T23:30:35Z")

</div>

Please look at links [7], [8], and [9] above for some basic information on grid cells.

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [January 31, 2018, 6:00am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/24 "2018-01-31T06:00:42Z")

</div>

I believe everybody is familiar with Mosers’ works here, but you mentioned it as a universal data structure format, however, to the best of my knowledge, it’s only a part of the navigation system. Do you have another vision of it?

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [January 31, 2018, 1:27pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/25 "2018-01-31T13:27:12Z")

</div>

I do.

Expand your view of navigation.

We know from patient HM that without the hippocampus you lose your ability to encode your navigation through experience. It makes some sense to think that the hippocampus samples the grid structure and rearranges or encodes the data that is proximal to the temporal lobe to be learned as your autobiographical experience.

The cortex is still forming the “global workspace”[1] but the mapping of navigation in this workspace is not being encoded and stamped into the temporal lobe.

[1] CONSCIOUSNESS AND COGNITIVE ACCESS NED BLOCK; See figure 4.

> **[2008\_Aristotsoc.pdf](https://www.nyu.edu/gsas/dept/philo/faculty/block/papers/2008_Aristotsoc.pdf)**
>
> 753.20 KB

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [January 31, 2018, 6:40pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/26 "2018-01-31T18:40:08Z")

</div>

This could be the where versus the what pathways of vision. If I am a fish and believe I am in a dangerous location and something appears to the right I know to fish fast to the left. On the other hand if I believe mates are near I choose to proceed in a courtly manner to the right.

For detailed “what” saccading to center the region of interest makes sense. Orientation is covered by both Hawkins’ theory and Hinton’s capsule networks. Hawkins being more biologically near and generalize-able.

Something when a thing is highly rotated I rotate my head to better see it, not something a mature adult does due to vanity.

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [January 31, 2018, 6:41pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/27 "2018-01-31T18:41:02Z")

</div>

I also appreciate your informative answer above. Can you please explain “distributed grids”.? Thanks.

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [January 31, 2018, 7:44pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/28 "2018-01-31T19:44:22Z")

</div>

> [@Bitking](#):
>
> the hippocampus grid structure

It looks you are talking about place cells (let’s generalize the term for direction and other functionally described types), not grid cells, which are located in the medial entorhinal cortex. Place cells don’t have the grid structure.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [January 31, 2018, 7:51pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/29 "2018-01-31T19:51:26Z")

</div>

There is good reason to believe that the place cells sample the grid cells with SDRs to form a pocket of response.

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [January 31, 2018, 9:34pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/31 "2018-01-31T21:34:32Z")

</div>

> [@Bitking](#):
>
> There is good reason to believe that the place cells sample the grid cells with SDRs to form a pocket of response.

Sure, it’s quite clear, that the grid cells provide an important part of the input to activate some place cells.  
I mean, grid cells are just one (even very important for real space navigation) of such inputs. Or, perhaps, it would be more clear to say, place cells can work without grid cells, but grid cells are useless without place cells.  
Place cells by themselves don’t have any grid or any other known to me structure. That’s why I was surprised to hear about universal data structure format in the brain.

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [January 31, 2018, 10:03pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/32 "2018-01-31T22:03:38Z")

</div>

This is an important question for both HTM and NNs. My understanding is that both do it in a disturbingly wasteful way. They both repeat detectors again and again all over the field and again and again at a different scale and again and again at a even coarser scale thousands of redundant detectors. Is there data that shows this is how biology works? pro side it is simple, con side it is wasteful.

I once heard a talk by someone from Columbia University on rate coding and how to use it to do transformations like scale and other things (placement, orientation?? not sure I remember this). If scale transformations are easy in rate coded spiking system it would be good to know that. One or a few detectors applied at many scale settings?

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [January 31, 2018, 10:22pm UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/33 "2018-01-31T22:22:48Z")

</div>

paper on vision in primates

> **[RajalinghamIssaDiCarlo\_20180101\_bioRxiv.pdf](https://issalab.neuroscience.columbia.edu/sites/default/files/content/RajalinghamIssaDiCarlo_20180101_bioRxiv.pdf)**
>
> 8.80 MB

Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 1, 2018, 12:00am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/34 "2018-02-01T00:00:55Z")

</div>

Spin,

The entorhinal cortex is cortex. One of the takeaway lessons I have gotten from books like “On Intelligence” is that cortex is cortex. The distributed grid representation is clearly spread over a very large chunk of the entorhinal cortex. Why should this computing pattern not be found elsewhere in the cortex?

The older allocortex in the hippocampus communicates with the neocortex and distills this representation into chunks like “places.” That strongly suggests that the place information is in the cortex distributed over this grid pattern to be sampled. Hence the claim of a distributed representation in the grid cells. SDRs are perfect to get a little taste of several different intermingled distributed representations from many sensory modalities and combine them into a meaning.

I have many reasons think that there are other codings like “place” in the hippocampus but that they are not as easy to stumble into by monitoring free-roaming critters. It may be a real stretch to suggest that other parts of the brain use a grid pattern but consider this: we were able to sample grid cells because there was a strong relationship to something happening in the cage to correlate with. (See reference [7] above) I don’t think I have seen anyone looking for this sort of property in other parts of the cortex.

I find distributed representation in grid patterns to be a delightful solution to the coding problem. Even if this turns out to be non-biological for most of the cortex it codes very well in neural networks. The ratslam people are doing very interesting things with this. In the H of HTM, we need to get from a point in the input pattern space and spread it out to interact with other parts of the pattern - a distributed representation. Humans have trouble visualizing this as we see in 2.5 D. (2D with stereo cues & color) Once we go from the highly topographic V1 cortex it gets hard to conceptualize how the data gets blended together. The V1 cells look like a topographic pattern because the retina feeds it that. Once we get into progressive visual maps it gets less clear what is happening to the representation. I can see how a sort of FFT transform could be spreading and combining different frequencies of sampled data across a distributed pattern in these maps. By the time we get to the association areas we have multiple streams of senses blended together. What is the lingua franca? Grids would work as well as anything here.

I agree that this is highly speculative and that it may be premature to assert that this is the ground truth but it does fit very well with many years of reading about this sort of thing - it just feels right. Win or lose I will be researching to attempt to learn if this is how the brain does it. I strongly suspect that the answer will be this or something very close.

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [February 1, 2018, 4:15am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/35 "2018-02-01T04:15:42Z")

</div>

Bitking super interesting post. I expect it will take me years to understand. Thanks.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 1, 2018, 4:43am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/36 "2018-02-01T04:43:48Z")

</div>

It took me years. On the other hand - nobody showed me what to look for.

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [February 1, 2018, 5:08am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/37 "2018-02-01T05:08:05Z")

</div>

> [@Bitking](#):
>
> Why should this computing pattern not be found elsewhere in the cortex?

Frankly, I can’t find the purpose of using the periodical grit everywhere in the cortex. I was charmed by the hexagon magic of grid cells too, but after spending reasonable time in thinking about it, I didn’t find any use for them outside the of creating a virtual space, what is super important for _some_ modalities _on the lowest levels of hierarchy_, but how could it be useful in all other cases?  
BTW, even the founder of the grid cell Edvard Moser, who continues his work in this field, quite often speaks about the importance of place cells for the memory mechanics, but not grid cells (he also sad the idea of FFT as the key mechanics for grid cells hadn’t been confirmed).  
I agree, that we are still on the early stage of collecting facts in this field and it’s very complicated and time-consuming to conduct research here, so some important information can be omitted. However, how the brain could use such grid as a general base for everything else? Perhaps, you could provide a simple imagining example for such mechanics for some nonspatial patterns to get the flavour of your vision?

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 1, 2018, 6:02am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/38 "2018-02-01T06:02:46Z")

</div>

Consider the entorhinal cortex and what is being represented: a mixture of the vestibular system, visual representation, self-motion, postural feedback. Perhaps more. When you look at the repeating patterns where do you see any of that in the distributed dispersal that we choose to call a grid system?

I see that as the output of a common digestion system that ends up in a “common format.” The grid cells under discussion are a byproduct of explaining the coding that surrounds the hippocampus. Even though there are some very interesting theories to be best of my knowledge nobody really is sure of how that grid pattern is formed.

Looking at the dizzying maze of fiber tracts that connect the maps in the brain I have often wondered about the palimpsest of patterns that lay one on top of another in these many tracts. Consider the areas that receive the foveal vision as the eye darts over a scene, bit by bit - color, texture, stereo disparity, edges - all overlapping on the same maps. Within a few layers of maps that stew is mixed with the digested versions of body sensors (vibration, temperature, joint position and muscle stretch) to guide end-effector motion in space in the association areas.

The same general thing is going on in the auditory tracts.

Fat fiber bundles connect these diverse association areas and communicate some sort of meaningful information.

It gives me hope that if we can tease out how the wildly dissimilar information is combined in the grid maps that perhaps the same general process is working in everywhere in the brain.

You call grids a specialized navigation system - I see it as a possible general coding scheme that has been demonstrated to work in part of the brain. I will say it again - if the cortex is a general fabric that is uniform throughout then working out how part of it functions could apply to all of it.

Ask yourself - if you were going to take all of these wildly diverse sensory organs and apply a common process to turn it into a maximally dispersed (and compatible) coding to gain the best resistance to decay over time - what format would you expect it to end up as? For bonus points use a fabric of HTM style columns that do local processing to do this job.

---

<div class="post-metadata">

**Author:** ![spin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/spin/32/1539_2.png) [@spin](https://discourse.numenta.org/u/spin)\
**Post date:** [February 3, 2018, 1:09am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/39 "2018-02-03T01:09:06Z")

</div>

> [@Bitking](#):
>
> I see it as a possible general coding scheme that has been demonstrated to work in part of the brain

I really support everything you’ve just sad, but I see generalized place cells in this role, not grid cells. Again, what could it be the reason to use the 2D grid for auditory modality and how can it be applied on abstract levels like the meaning of the speech?

---

<div class="post-metadata">

**Author:** ![Ed\_Pell](https://avatars.discourse-cdn.com/v4/letter/e/6bbea6/32.png) [@Ed\_Pell](https://discourse.numenta.org/u/Ed_Pell)\
**Post date:** [February 3, 2018, 2:59am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/40 "2018-02-03T02:59:40Z")

</div>

Spin, because hearing is binocular. We need to place a sound on a 2D grid left/right and up/down. Exactly as we map vision onto a 2D grid.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 3, 2018, 3:15am UTC](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051/41 "2018-02-03T03:15:28Z")

</div>

To amplify this point: a next level of representation, speech sounds, can be arranged in a 2-d mapping arrangement.  
[http://www.animations.physics.unsw.edu.au/jw/voice.html](http://www.animations.physics.unsw.edu.au/jw/voice.html)

[Previous page](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051.md?page=1)

[Next page](https://discourse.numenta.org/t/how-htm-is-supposed-to-deal-with-spatial-invariance/3051.md?page=3)