# Reinforcement Learning and HTM Algorithm

**URL:** <https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965>\
**Category:** Machine Learning\
**Tags:** sequence-memory, encoders, question, community, nupic\
**Created:** [May 6, 2019, 9:12am UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965 "2019-05-06T09:12:38Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![Gary\_Gaulin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/gary_gaulin/32/2248_2.png) [@Gary\_Gaulin](https://discourse.numenta.org/u/Gary_Gaulin)\
**Post date:** [June 17, 2019, 3:25am UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/23 "2019-06-17T03:25:49Z")

</div>

> [@Bitking](#):
>
> The good or bad is stored as a property like a color. This goes with the concept that an object is a collection of features. In this case - the feature is some signal communicated by the amygdala.

While writing I thought about getting onto the amygdala related detail, but did not want to get wordy or make you have to wait to get more just right.

The amygdala would be an emergency type fast response parallel memory subsystem where good or (especially) bad can be stored as a property.

What gets adjusted each time a given action is tried is in this case a non-amygdala confidence level that changes depending on how many times a given action worked and how many times it was tried.

Something never experienced before would have recall of nothing. In that case we right away know we have to take at least a random guess then save data representing whether it went good or bad after trying. After working a few or more times in a row confidence level increases to a point where if it did not for some reason work our confidence would go down a little but we would then still try again before giving up on it.

All the rest of the associated memory data can remain exactly the same.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [June 17, 2019, 4:35am UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/24 "2019-06-17T04:35:21Z")

</div>

> [@Gary\_Gaulin](#):
>
> The amygdala would be an emergency type fast response parallel memory subsystem where good or (especially) bad can be stored as a property.

The amygdala is full of hardwired archetypes (genetically programmed) that are evaluated at a subconscious level. These programmed features are things like snakes, spiders, features of your own species, faces, secondary sexual characteristics, and the like. This is fed from a tap on the feed to the visual system and is mostly a one-way feed into the hippocampus.

This is the source of much of the good/bad signals.

More on the amygdala processing here:

> [@Thoughts on multiple pathways for sensory input into the neocortex](https://discourse.numenta.org/t/thoughts-on-multiple-pathways-for-sensory-input-into-the-neocortex/3129/3):
>
> It seems to be a thing for connectome/cortex researchers to ignore the processing that goes on in the “lower” brain structures. These were/are sufficient to run lizards and amphibians. There is absolutely no reason to think that they re not performing much of their original functions inside of the functioning brain. The lower brain receives a low-resolution image from the eye area “away” from the fovea and directs the eye to point at interesting areas to form identification of an object with a …

Some of the other good/bad features come from other limbic system nodes such as the hypothalamus.

I think that there may be a mechanism for adding new features to the amygdala based on the utility this function would add but I am unaware of any mechanism to do so.

---

<div class="post-metadata">

**Author:** ![Gary\_Gaulin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/gary_gaulin/32/2248_2.png) [@Gary\_Gaulin](https://discourse.numenta.org/u/Gary_Gaulin)\
**Post date:** [June 17, 2019, 6:59am UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/25 "2019-06-17T06:59:35Z")

</div>

> [@Bitking](#):
>
> The amygdala is full of hardwired archetypes (genetically programmed) that are evaluated at a subconscious level. These programmed features are things like snake shape, spiders, features of your species, faces, secondary sexual characteristics, and the like. This is fed from a tap on the feed to the visual system and is mostly a one-way feed into the hippocampus.

Yes. The way I see it: at the “subconscious level” other examples would be where hand is too close to flame and amygdala kicks in to quickly recoil back with fear the amygdala state is enough to make sure confidence in that action working gets lowered due to something having gone wrong from getting that close. We did not have to learn that sudden pain like that is bad as though there is choice over responding or not then take time thinking about it, the amygdala already knows it’s bad and immediately responds with fast reflex in other direction.

When something bad looking jumps in our way we can get startled by fear, without our at the logical level deciding whether we should be or not.

A one-way feed into the hippocampus is what the amygdala would need to have some control over what gets mapped in as the attractor to navigate towards. As a result “love is blind”, will “walk miles for” and all that.

While traveling: as long as everything that could go wrong didn’t the confidence in an action working increases (or remains at max value).

In the [ID Lab](https://discourse.numenta.org/t/oscillatory-thousand-brains-minds-eye-for-htm/3726) model confidence goes down from bashing into solid objects, shocks to feet and navigational errors like heading not matching the spatial map given angle and magnitude. For at least us an amygdala adds extra feeling to confidence changing experiences, but for neocortical modeling purposes a feedback bit representing an “Ouch!” or “Oh crap!” from any system that can sense one is enough to know when something just went wrong somewhere.

The HTM part would be a neocortical sensory to sparsely addressed parallel memory made of virtual neurons, instead of sensory to densely addressed digital RAM made of silicone that is not at all parallel but similarly works well enough for ~28 or less address bits to have been worth experimenting with that over the years.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [June 17, 2019, 3:42pm UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/26 "2019-06-17T15:42:51Z")

</div>

> [@Gary\_Gaulin](#):
>
> Yes. The way I see it: at the “subconscious level” other examples would be where hand is too close to flame and amygdala kicks in to quickly recoil back with fear the amygdala state is enough to make sure confidence in that action working gets lowered due to something having gone wrong from getting that close. We did not have to learn that sudden pain like that is bad as though there is choice over responding or not then take time thinking about it, the amygdala already knows it’s bad and immediately responds with fast reflex in other direction.

This is not a good example - fast reflex due to pain is handled in the spine.

> **[Reflex arc](https://en.wikipedia.org/wiki/Reflex_arc)**
>
> A reflex arc is a neural pathway that controls a reflex. In vertebrates, most sensory neurons do not pass directly into the brain, but synapse in the spinal cord. This allows for faster reflex actions to occur by activating spinal motor neurons without the delay of routing signals through the brain. The brain will receive the input while the reflex is being carried out and the analysis of the signal takes place after the reflex action.
> There are two types: autonomic reflex arc (affecting inner o...

I cannot say this enough - the nervous system is a very complicated machine with many interacting systems. Putting a given function in the right system(s?) has taken me many years and I still get it wrong from time to time.

Localizing a function is can be very difficult as different aspects of even the simplest thing are likely to be shared over several systems at several levels of processing. Stating that “x does y” is likely to miss important parts of how the system functions.

The amygdala is often described as fear processing with little regard to the other things it can recognize and process. (and tag in the hippocampus) From an evolutionary point of view, this makes good sense - being afraid of everything is a good call in an uncertain world; almost anything can kill you.

The amygdala also recognizes other things like social cues in faces and secondary sexual characteristics. As much as some people would like it to be different - you do judge people by how likely it seems that they have the right stuff to procreate with you. Your ability to read expressions is unbelievably sensitive and built in at a sub-cortical level. Your processing of social cues such as dominance and submission, likewise.

This is all baked in by evolution and it takes considerable social education to overcome it. I think that recognizing the innate workings of the brain and designing social structures to work with this rather than suppress these natural drives would result in better-adjusted people. And fewer rogue humans.

---

<div class="post-metadata">

**Author:** ![MaxLee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/maxlee/32/5546_2.png) [@MaxLee](https://discourse.numenta.org/u/MaxLee)\
**Post date:** [June 17, 2019, 8:51pm UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/27 "2019-06-17T20:51:26Z")

</div>

I recently came across a TED talk about the importance of sleep. One of the takeaways I got from it was that sleep seems to be a process by which we’re moving learned information from short-term into long-term memory, perhaps by dreaming(?). This could even be modified/amplified by introducing very small impulses to the outside of the skull.

Within the concept of “replay”, this at least makes sense; short-term memory is relaying what it learned to the other parts of the brain by replaying bits and pieces of it, which the magic of our brains is then able to organize and store into long-term knowledge.

Certainly encourages me to sleep more, and consider that perhaps we should have columns occasionally playing back out to each other, somehow.

> **[Sleep is your superpower](https://www.ted.com/talks/matt_walker_sleep_is_your_superpower)**
>
> Sleep is your life-support system and Mother Nature's best effort yet at immortality, says sleep scientist Matt Walker. In this deep dive into the science of slumber, Walker shares the wonderfully good things that happen when you get sleep -- and the...

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [June 17, 2019, 8:57pm UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/28 "2019-06-17T20:57:10Z")

</div>

These two posts are on data coding and the role of sleep, and the related transfer between the hippocampus and cortex.

> [@How does short term memory work?](https://discourse.numenta.org/t/how-does-short-term-memory-work/4420/10):
>
> You did ask about LT/ST outside of the hippocampus and I did not answer your question directly but pointed you to the central nature of the [entorhinal cortex](https://en.wikipedia.org/wiki/Entorhinal_cortex) / [hippocampal formation](https://en.wikipedia.org/wiki/Hippocampal_formation) (EC/HF) in the memory systems. I feel like I left your question less than half answered. There are several interrelated time scales in the memory system. The shortest is actual perception. Say what? Yes - perception is an active recall process. In the first part of perception, cells in the cortex and thalamus ha…

And this one that talks a little bit about the actual transfer:  
_… brain wave are known to excite the hippocampus in dreaming - driving a recall in both the cortex and hippocampus at the same time. If the hippocampus has leaned a response that makes it fire faster from new learning (using the “standard” spike timing learning ) this could trigger learning in the related cortex. This excitation sweeps through again and again as long as there are significant phase differences in response between the two areas."_

> [@On the necessity of sleep](https://discourse.numenta.org/t/on-the-necessity-of-sleep/4860/9):
>
> To add a little more background on the nature of what is being encoded: I believe that everything that you learn is framed and parsed in terms of what you have learned before; you learn delta coding based on your prior life experiences. If you extend this all the way back you build on your innate built-in instinctive and emotional wiring as dictated by your genetics. It is nature first, then nurture. To flesh this out - as you experience the world prior memories are being recalled as part of y…

---

<div class="post-metadata">

**Author:** ![Gary\_Gaulin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/gary_gaulin/32/2248_2.png) [@Gary\_Gaulin](https://discourse.numenta.org/u/Gary_Gaulin)\
**Post date:** [June 18, 2019, 12:51am UTC](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965/29 "2019-06-18T00:51:26Z")

</div>

> [@Bitking](#):
>
> This is not a good example - fast reflex due to pain is handled in the spine.

Thanks for catching that one.

My example needs to include the automatic spinal reflex motor action recoil, which in turn alerts the amygdala something bad we need to fear just happened. There are two independent subsystems at work, instead of one.

I’m not used to having to include even spinal related detail.

[Previous page](https://discourse.numenta.org/t/reinforcement-learning-and-htm-algorithm/5965.md?page=1)