# Space is a latent sequence

**URL:** <https://discourse.numenta.org/t/space-is-a-latent-sequence/10334>\
**Category:** Tangential Theories\
**Created:** [December 17, 2022, 3:09am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334 "2022-12-17T03:09:10Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kroggen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/kroggen/32/6405_2.png) [@kroggen](https://discourse.numenta.org/u/kroggen)\
**Post date:** [December 17, 2022, 3:09am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/1 "2022-12-17T03:09:10Z")

</div>

Paper from DeepMind:

**Space is a latent sequence: Structured sequence learning as a unified theory of representation in the hippocampus**

> **[2212.01508.pdf](https://arxiv.org/pdf/2212.01508.pdf)**
>
> 6.05 MB

> … we provide a unifying principle that the mental representation of space is an emergent property of latent higher-order sequence learning. Treating space as a sequence resolves myriad phenomena, and suggests that the place-field mapping methodology where sequential neuron responses are interpreted in spatial and Euclidean terms might itself be a source of anomalies. Our model, called Clone-structured Causal Graph (CSCG), uses a specific higher-order graph scaffolding to learn latent representations by mapping sensory inputs to unique contexts. Learning to compress sequential and episodic experiences using CSCGs result in the emergence of cognitive maps - mental representations of spatial and conceptual relationships in an environment that are suited for planning, introspection, consolidation, and abstraction. We demonstrate that over a dozen different hippocampal phenomena, ranging from those reported in classic experiments to the most recent ones, are succinctly and mechanistically explained by our model.

Maybe the missing piece for HTM

This can be revolutionary!

---

<div class="post-metadata">

**Author:** ![EEProf](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/eeprof/32/5653_2.png) [@EEProf](https://discourse.numenta.org/u/EEProf)\
**Post date:** [December 17, 2022, 11:51am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/2 "2022-12-17T11:51:13Z")

</div>

Nice find. Thanks!

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [December 17, 2022, 9:41pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/3 "2022-12-17T21:41:32Z")

</div>

One of the authors - [Dileep George](https://en.wikipedia.org/wiki/Dileep_George) - is co-founder of Numenta.  
And this paper is in the vein of his previous research/papers done at Vicarious, before joining DeepMind.

PS Everything, not only space-mapping is based on the same sequence-stitching algorithm.

PS2 here-s an earlier, related presentation [space is a sequence - YouTube](https://www.youtube.com/watch?v=rFhzIL9_y5w)

---

<div class="post-metadata">

**Author:** ![david.pfx](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@david.pfx](https://discourse.numenta.org/u/david.pfx)\
**Post date:** [December 17, 2022, 11:31pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/4 "2022-12-17T23:31:26Z")

</div>

Sorry, hour and a half is too long. Is there a tldr? Published paper?

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [December 17, 2022, 11:32pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/5 "2022-12-17T23:32:58Z")

</div>

yes check papers on dileep george. Including the one in the initial message. The video helps make sense of the idea if the paper wasnt sufficient. Also there-s a shorter clip on the yt channel

The TLDR is Alphabet found the content interesting enough to acquire Vicarious, after investing much more time & attention than you want to spend

---

<div class="post-metadata">

**Author:** ![CollinsEM](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/collinsem/32/160_2.png) [@CollinsEM](https://discourse.numenta.org/u/CollinsEM)\
**Post date:** [December 18, 2022, 3:22pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/6 "2022-12-18T15:22:10Z")

</div>

As a graph model, this model seems fairly straight forward. His description does appear to invoke a temporal memory algorithm similar to what we are familiar with from HTM.

As a physicist, I can’t help but draw the connection to a position-momentum phase space plot from dynamical systems theory.

![position-momentum-phase-space](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/5/5312aa196ccbc964ab8a842df2e2a703c162c18a.png)

Here, each node in the graph encodes both position and velocity. Each trajectory should generate a unique path through this phase space (e.g. forward vs. backward). The only ambiguity that might result in a bifurcation of paths would be when the agent decides to change direction or keep going while passing through the zero momentum state (middle path above). However, even this might be resolved with a third axis to the space representing orientation.

It should be possible to discretize this space with a neural representation if position, orientation, and velocity are each provided as independent inputs (or can be inferred from the input sensory stream). Acceleration might also be measured by the proprioceptive system or inferred from motion cues. With these inputs, it should be possible to track each unique path through this phase space and allow for disambiguation of current states and prediction of future states by position (where am I), orientation (what direction am I facing), duration (how long do I expect to remain in this location).

There may be evidence for the generation unique place cell representations based on orientation and/or direction of travel. I’ve heard of an experiment that was measuring place cells in rats that were tasked with traveling back and forth down a corridor. Place cells associated with specific locations were observed. However, over time, some of the place cells began to only respond while traveling through a location in a specific direction. On the return trip, another place cell was observed to be active in the same location.

---

<div class="post-metadata">

**Author:** ![EEProf](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/eeprof/32/5653_2.png) [@EEProf](https://discourse.numenta.org/u/EEProf)\
**Post date:** [December 19, 2022, 2:32pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/7 "2022-12-19T14:32:09Z")

</div>

> [@CollinsEM](#):
>
> As a graph model, this model seems fairly straight forward.

Consider a Petri Net.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [December 19, 2022, 3:24pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/8 "2022-12-19T15:24:44Z")

</div>

> [@EEProf](#):
>
> Consider a Petri Net.

Ewwwww!  
What is that stuff?

---

<div class="post-metadata">

**Author:** ![Prakash](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/prakash/32/6117_2.png) [@Prakash](https://discourse.numenta.org/u/Prakash)\
**Post date:** [December 24, 2022, 11:43am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/9 "2022-12-24T11:43:17Z")

</div>

![Untitled18](https://canada1.discourse-cdn.com/flex030/uploads/numenta/original/2X/b/b65ca1bd1852553498cbd1c1990287eddddd4469.png)

Space is a latent sequence because of this mechanism. Just-knowing and prediction mechanism works together, which perceived as sequence.

---

<div class="post-metadata">

**Author:** ![SeanOConnor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/seanoconnor/32/3441_2.png) [@SeanOConnor](https://discourse.numenta.org/u/SeanOConnor)\
**Post date:** [January 16, 2023, 4:55am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/10 "2023-01-16T04:55:44Z")

</div>

There are if-except-if trees. If some test is true then return a response value, except if a further test is true then return the associated response value, except if a further test is true then …  
That way you can go from general to more specific responses to an input.  
It is probably better to use n-way tree nodes than just the binary ones in the example.

---

<div class="post-metadata">

**Author:** ![cezar\_t](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cezar_t/32/4451_2.png) [@cezar\_t](https://discourse.numenta.org/u/cezar_t)\
**Post date:** [January 19, 2023, 11:43am UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/11 "2023-01-19T11:43:45Z")

</div>

> [@CollinsEM](#):
>
> There may be evidence for the generation unique place cell representations based on orientation and/or direction of travel. I’ve heard of an experiment that was measuring place cells in rats that were tasked with traveling back and forth down a corridor. Place cells associated with specific locations were observed. However, over time, some of the place cells began to only respond while traveling through a location in a specific direction. On the return trip, another place cell was observed to be active in the same location.

I think their main point is there is no explicit separate structures/algorithms for place/grid cells (or reference frames). The agents uses only “strips of experience” to map using movement (action-\>observation-\>action…) their territory, being it either physical or conceptual or _mixed_, by generating a knowledge graph.

_And that’s all what is happening_ in the agent’s brain.

What they also observed - and claim - is that whatever nodes become active in the above process parallels activations of place/grids cells so whatever observation lead to “discovery” of grid cells was only the “noise” made by the function of their proposed mechanism - Clone Structured Causal Graph.

PS Even the (mental representation aka model of) space itself emerges from learned strips of experience, place/grid cells being artifacts of how this model is encoded

---

<div class="post-metadata">

**Author:** ![SeanOConnor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/seanoconnor/32/3441_2.png) [@SeanOConnor](https://discourse.numenta.org/u/SeanOConnor)\
**Post date:** [January 23, 2023, 12:52pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/12 "2023-01-23T12:52:26Z")

</div>

One example of if except if trees is if the previous letter (at n-1) is “a” then the current letter (at n) should be “s” except if the previous previous letter (at n-2) is “c” then the current letter should be “t”.  
Then you have memorized “as” and “cat”. And of course you can memorize a billion things that way, using a simple greedy learning algorithm.  
That is imperfect because the further back you go in a sentence the less informative the letter tests are.  
You could turn an image into a stream of symbols using a random projection and binarization and then all the symbols have about equal information. I don’t know if that would get you to neural network level performance but it is something to think about.

---

<div class="post-metadata">

**Author:** ![BrainVx](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@BrainVx](https://discourse.numenta.org/u/BrainVx)\
**Post date:** [January 26, 2023, 10:03pm UTC](https://discourse.numenta.org/t/space-is-a-latent-sequence/10334/13 "2023-01-26T22:03:37Z")

</div>

The CSCG paper was in a previous post back in May 22

> [@Clone-structured cognitive graphs (CSCG)](https://discourse.numenta.org/t/clone-structured-cognitive-graphs-cscg/9739):
>
> I do not know if everyone is familiar with Dileep George’s paper [Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps](https://www.nature.com/articles/s41467-021-22559-5). In any case, he recently gave a seminar at the Simons Institute [Space is a latent sequence: a unifying theory of representations and remapping in the hippocampus](https://archive.org/details/Redwood_Center_2022_05_18_Dileep_George) where he presented the main concepts in this paper. According to Dileep, Clone-structured cognitive graphs (CSCG)s offer a coherent explanation for place cells, pl…