# Difference between SMI and RL?

**URL:** <https://discourse.numenta.org/t/difference-between-smi-and-rl/6752>\
**Category:** Lounge\
**Created:** [October 24, 2019, 3:25pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752 "2019-10-24T15:25:46Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [October 24, 2019, 3:25pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/1 "2019-10-24T15:25:46Z")

</div>

I’m writing my graduation project report and got confused. What is the difference between SMI (Sensor-Motor Inference) and RL (Reinforcement Learning)? Can I assume SMI to be HTM’s version of RL?

Thanks

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [October 24, 2019, 3:43pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/2 "2019-10-24T15:43:54Z")

</div>

SMI does not include reward / punishment. SMI can simply be a model that accepts sensory input and incorporates self action into its model of an environment over time.

RL requires a function that calculates some reward or punishment (usually) based upon a model, then makes a decision upon what action to take next.

A RL system could use an SMI model.

---

<div class="post-metadata">

**Author:** ![CollinsEM](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/collinsem/32/160_2.png) [@CollinsEM](https://discourse.numenta.org/u/CollinsEM)\
**Post date:** [October 24, 2019, 4:29pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/3 "2019-10-24T16:29:30Z")

</div>

In my own taxonomy of machine learning techniques, the main distinction that I draw between the two relates to how much a’priori knowledge, or meta-understanding of the system, is available to guide the learning of the agent.

Reinforcement Learning is usually considered a supervised (or semi-supervised) learning technique. This typically implies that that the reward function (or cost) function is known in advance or there is some form of a supervisor (e.g. human or GAN) available that knows the difference between right and wrong.

Sensor-motor Inference, on the other hand, is generally considered to be completely unsupervised. In SMI, the agent is attempting to make predictions about what to expect from its sensory input based on external environmental factors as well as its interactions with the environment (including its own movement and placement of its sensors). The agent adjusts its internal representation based on the deviation of the prediction from the observed sensory input. Sometimes this is due to an insufficient model of the environment and/or self-interactions with the environment. Sometimes it’s due to the surprise of novel phenomena or interactions. Ultimately, the agent learns to adapt to whatever behavior the environment exhibits. If there is sufficient regularity to this behavior, then the predictions of the agent should improve over time as its internal representation is refined.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [October 24, 2019, 6:33pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/4 "2019-10-24T18:33:50Z")

</div>

How do you think a captured training signal fits in your scheme?  
In the dad’s song project we posit that the structure of language is learned by listening first, then imitation.

_In a nutshell:  
Passively learning dad’s song coupled with emotional flavoring.  
Passively learning with other sounds with no or other emotional flavoring.  
(Could be alert calls or other signalling such as food)  
Active learning to produce sounds by exploratory vocalizing motor skills.  
Time passes.  
Hormonal / reinforcement driven learning to create prior learned Dads song._

---

<div class="post-metadata">

**Author:** ![Falco](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Falco](https://discourse.numenta.org/u/Falco)\
**Post date:** [October 24, 2019, 7:08pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/5 "2019-10-24T19:08:32Z")

</div>

This turns out to be a very interesting question. There isn’t even a wiki about Sensor-Motor Inference.

> [@CollinsEM](#):
>
> In SMI, the agent is attempting to make predictions about what to expect from its sensory input based on external environmental factors as well as its interactions with the environment (including its own movement and placement of its sensors). The agent adjusts its internal representation based on the deviation of the prediction from the observed sensory input.

Could we try to define a precise definition for this? Where does it start and end? Maybe define a flow chart using HTM terminology?

How could we compare it to an affordance (Cisek’s term)? Is an SMI a set of affordances? Or are there different SMIs?

---

<div class="post-metadata">

**Author:** ![CollinsEM](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/collinsem/32/160_2.png) [@CollinsEM](https://discourse.numenta.org/u/CollinsEM)\
**Post date:** [October 25, 2019, 2:43pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/6 "2019-10-25T14:43:07Z")

</div>

> [@Bitking](#):
>
> How do you think a captured training signal fits in your scheme?

I’m not familiar with this concept.

I’ve also not considered the behavioral aspects of imitation to be a direct consequence of sensory-motor inference. Such behavior may not be adequately explained by the over-simplified model of SMI that I outlined above. But then again, I was only trying to draw a distinction betwen RL and SMI as requested by the OP.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [October 25, 2019, 9:29pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/7 "2019-10-25T21:29:30Z")

</div>

Please see this paper (page 444) for a description of latent learning that is expressed after a very long interval:  
**The Evolution of Multiple Memory Systems - David F. Sherry, Daniel L. Schacter**

> **[\[PDF\] The Evolution of Multiple Memory Systems - Semantic Scholar](https://www.semanticscholar.org/paper/The-Evolution-of-Multiple-Memory-Systems-Sherry-Schacter/6a22534a692182995e54d34f932cc6bf3d6adfb1)**
>
> The existence of multiple memory systems has been proposed in a number of areas, including cognitive psychology, neuropsychology, and the study of animal learning and memory. We examine whether the existence of such multiple systems seems likely on...

or

[https://psycnet.apa.org/doiLanding?doi=10.1037%2F0033-295X.94.4.439](https://psycnet.apa.org/doiLanding?doi=10.1037%2F0033-295X.94.4.439)

This latent learning seems to be related to imprinting.

This learned template forms a strong training signal that is as immediate during the SMI as tactile feedback. It is of a different nature but I see the basic mechanism as being very similar; the error signal is closely coupled to the motor performance.

---

<div class="post-metadata">

**Author:** ![Falco](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Falco](https://discourse.numenta.org/u/Falco)\
**Post date:** [October 25, 2019, 10:28pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/8 "2019-10-25T22:28:20Z")

</div>

This is interesting, but it brings up a few questions.

- Since birds didn’t evolve the same structure as our neocortex, does that mean that SMI omits the neocortex?

- Bird song SMI only involves the vocal chords (or whatever a bird uses to sing) and auditory feedback. Isn’t the SMI supposed to move the subject in the world, and thereby change its world model?

To me, the inference part means that every output to a muscle, coming from the neocortex, receives a direct feedback from the muscle to the emitting zone, combined with one or more secondary inputs that help rectifie the output.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [October 25, 2019, 10:40pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/9 "2019-10-25T22:40:40Z")

</div>

This is why I bring this up - the connection is somewhat more distant than the direct connections across the sylvian fissure. (Motor to somatosensory cortex)

In this case, we have some internal model that is compared to the perceived sound - the difference being the error signal.

With SMI you have to consider that the connection between perception and some movement still requires some feedback from some other part of the brain saying that the motion is doing something wanted in guided motion. These are two slightly different activities.

I don’t think it takes much imagination to think of humans trying to imitate a perceived and remembered sound. You can see babies babbling to learn to run the voice mechanism, and then a distinct difference when they try to make some remembered sound. Again, two slightly different activities.

Babies are a wonderful laboratory to see learning in action if you pay attention.

---

<div class="post-metadata">

**Author:** ![Falco](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Falco](https://discourse.numenta.org/u/Falco)\
**Post date:** [October 25, 2019, 10:58pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/10 "2019-10-25T22:58:12Z")

</div>

I see.

There is another difference though, I think.

My brother is a physiotherapist, and he told me that people who sprayed an ankle run a higher risk of spraying it again, because they need to relearn to walk after their cast is taken off. It sounds like during wearing the cast, their _SMI_ was modified to walk differently, and they _forgot_ how to use the muscles properly.

So this means the learning is continuous and constantly adaptive to new situations.

> [@Bitking](#):
>
> Babies are a wonderful laboratory to see learning in action if you pay attention.

I should get me one of those. Are they expensive? ;-).

---

<div class="post-metadata">

**Author:** ![jimmyw](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/jimmyw/32/1485_2.png) [@jimmyw](https://discourse.numenta.org/u/jimmyw)\
**Post date:** [October 26, 2019, 10:09am UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/11 "2019-10-26T10:09:35Z")

</div>

> [@Bitking](#):
>
> Babies are a wonderful laboratory to see learning in action if you pay attention.

Agreed, had a newborn in the house at the time I was first getting into HTM. Really useful watching them develop while you’re mulling over the neuroscience.

> [@Falco](#):
>
> My brother is a physiotherapist, and he told me that people who sprayed an ankle run a higher risk of spraying it again, because they need to relearn to walk after their cast is taken off. It sounds like during wearing the cast, their _SMI_ was modified to walk differently, and they _forgot_ how to use the muscles properly.

I experienced exactly this with a soccer injury, an ankle sprain. Re-injured it every subsequent season until I found a good physio, who explained why proper rehab was important for this very reason.

The physical injury will always heal itself, but if you don’t give the brain enough practice re-calibrating with the muscles though various daily balancing exercises, then you won’t be able to keep balance during the awkward moments and that’s when you get injured again.

> [@Falco](#):
>
> I should get me one of those. Are they expensive? ;-).

They involve a significant investment of your time. Much easier to borrow one from a friend or family member for short periods.

---

<div class="post-metadata">

**Author:** ![marty1885](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/marty1885/32/1891_2.png) [@marty1885](https://discourse.numenta.org/u/marty1885)\
**Post date:** [October 26, 2019, 3:27pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/12 "2019-10-26T15:27:50Z")

</div>

> [@jimmyw](#):
>
> They involve a significant investment of your time. Much easier to borrow one from a friend or family member for short periods.

I love these comments on the forums. They are soooo to the heart and made my day that much better. 🥰

Anyone want to make one using HTM? 😛

---

<div class="post-metadata">

**Author:** ![CollinsEM](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/collinsem/32/160_2.png) [@CollinsEM](https://discourse.numenta.org/u/CollinsEM)\
**Post date:** [October 30, 2019, 9:15pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/13 "2019-10-30T21:15:02Z")

</div>

> [@Falco](#):
>
> Isn’t the SMI supposed to move the subject in the world, and thereby change its world model?

SMI does not necessarily require the agent to move in the world. Only that the agent is able to perform an action that has some affect on the environment. SMI then learns to recognize these environmental changes due to self-action as distinct from other changes due to external causes.

It may also be possible for these same (or similar) systems to build up a model of the interactions of other agents with the environment. If the other agents (or their behaviors) are sufficiently similar to the observing agent’s, then some of its self-interaction modeling could be applied towards this task. This may then be a part of the cognitive capabilities that lead to behaviors like imitation.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [October 30, 2019, 9:27pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/14 "2019-10-30T21:27:54Z")

</div>

In the same vein, the fiber tracts from the frontal lobe back to the sensory areas mean that you can perceive some of the actions wholly internally without passing out to the body and back through the senses.

While this might be hard to envision, with these connections, it should work that you can direct early “motor” activity to the higher-level object store to trigger recall that is perceived. These actions will be learned by the cerebellum much the same way as learning to walk, resulting in rapid and fluid mental sequences.

> [@QC at the Cerebellum](https://discourse.numenta.org/t/qc-at-the-cerebellum/4844/7):
>
> The Cerebellum is primarily connected to the motor cortex. Most think of this for the actuation of the body. I should mention that it also modulates the connections from the motor and pre-motor projections to various other sections of the brain and allow the motor neurons to act as as drivers for action in the brain, prompting new states that are eventually perceived by the the temporal lobe as what we usually call “thoughts.” The cerebellum facilitates the generation of longer and more stereo…

This is the same basic mechanism that I propose underlies the mechanism of consciousness.

> [@Intelligence vs Consciousness](https://discourse.numenta.org/t/intelligence-vs-consciousness/2595/41):
>
> My working theory on consciousness is fairly simple considering how complex the central nervous system is. Let me set the table on how I see the brain working: Focusing on the H of HTM. (Hierarchy) Consider this in a sensory stream general framework of (raw sense) to (collected sense/association) to (name/symbol) to (collected symbols/grammar) to (working memory) to (the emotionally colored here and now) in the tip of the temporal lobe. This is the stream of sensation that you know as your pe…

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [November 12, 2019, 4:58pm UTC](https://discourse.numenta.org/t/difference-between-smi-and-rl/6752/15 "2019-11-12T16:58:16Z")

</div>

2 posts were split to a new topic: [The definition of self](https://discourse.numenta.org/t/the-definition-of-self/6802)