Who/Where is the descision maker?

[mraptor]

CC algorithm so far covers the recording and recall and assumes receiving correct motor-comands/actions.
There is still “unused” layers and neurons that will probably add more functionality, but I don’t expect decision making to happen in the same CC.
Thats why I’m trying to figure out the Decision loop, only then we can have simple Agents doing useful tasks.

We can start with a simple tasks as object recognition OR path-finding.
Current CC model relies on L6L4-loop.
The process as I understand it is that we receive as

INPUT : Sensor information and Motor Action
OUTPUT : Location and Features (to Thalamus)

So far we have elaborated RECORDING device.
What we need next is Decision maker which would convert L&F to Action.

I know of two mechanisms to do that : Reinforcement Learning and Planing.
Lets concentrate in this post on RL.

What are the requirements of RL. At its most simplest a lookup table with the following format :

State:Action:Acummulated-discounted-reward

i.e. :

Loc&Future:Action:Q-value

In addition we need a GOAL : recognize object(Layer2/3 representation stabilizes ???) OR find the path (reach a destination landmark/Sense)

I have no idea of how the RL-loop is connected, does he uses the Basal-Ganglia.
Or it goes somewhere in the Cortex via Thalamus !

We have two problems :

 The Goal : In both cases of Obj-recognition and Path finding we need a Stored in memory SDR against which to compare.
   In case the Goal is unclear/fuzzy (i.e. we never seen this object) we need a separate process, lets ignore it for now.

 The Q-value : In traditional RL this is a Real number, but we cant use numbers here ! 

The Loc&Future => Action pair can probably be solved by simplifies TMem used as Classifier.
(In fact every time I tried implementing TM /3 times/ the CORE was a classifier which I then extend to TM, so this functionality comes for free)
The Goal could be just another transition MemorizedSDR => END-Action

The biggest problem I have is the Q-value … real numbers with precision are not suitable as SDR

So one solution is having Real value table outside of SDR loop.

The other option is a different mechanism ?

So take it from here. What are your proposals ?

PS> In this case I’m assuming we use RL when we have partial or no map or reference frame, we are exploring. Planning/Search would be when we know the steps but there is different paths. if you dont agree restate the problem in your way.

[Bitking]

See if this makes sense to you:

Basic system level diagram of the brain

HTM seeks to model and understand what is going on in your head. I tossed this drawing together from memory so it is sure to be missing some connections. I skipped the thalamic connections all together as this was already a very complicated drawing. Also not shown are the connections between the limbic system and the stream passing through the brain stem and the cerebellum. These are important but not necessary to illustrate the basic cortical/limbic data flows. The blue maps are cortex compose…

Low level processing in the old brain

I have posted several times offering that the lower-level control structures work something like a Boltzmann or Hopfield network. I would like to expand on this concept. Let’s start with the basics. In the brainstem processing, we have many body sensors that sample blood-based messengers such as osmotic concentration or glucose levels. All of the body sensations also pass through these systems and provide additional information on the outside environment. It is the job of these low-level contr…

[adam]

I also have thought about this question for very long, and have not found an easy answer! It helped me however to think, what movements are essentially trying to achieve. So basically why should a CC move at all? What motor output should it do next?

I have two examples, what L5 (I suspect L5 cells as motor output of CC) cells in a Column must be capable of:

• directed behaviour: I want to reach for XYZ. This is probably organized in a (messy) hierarchy, similar to how the body joints can be organized in a hierarchy. Because you cant move your finger to reach a cup, but moving solely the arm, does not solve the task either.
• exploration of objects: what type of cup holding in my hand? Is it my numenta coffee, or the my numenta where the handle broke off? Lets try to reach for the handle and then my new sensation allows me to be certain, which one of the two cups I am holding in my hand.

What I learned from robotics: If you have the goal to minimize uncertainty after movement, you get a pretty good policy to perform a directed behavior. I think we implement this somehow, when we have two models of similar objects in mind and we want to find out, which of those are we holding in our hand. We immediately reach for the feature at the location, where we expect a difference.

I think the cortical column is always doing a directed movement in particular direction, may it be for the goal of accomplishing a higher order goal or simply exploration of an object.
Basal ganglia are gating those movements, so they simply allow only one movement, the loudest one. But the lower level brain structures will not solve the task, where to move, because only the neocortex has the model of the world and can answer this question.

So maybe the cortex is doing the decision making at column level and the basal ganglia / other old brain parts are deciding which output of the neocortex makes most sense (global level). This way you avoid to search for a central decision maker which I think more closely captures the behavior which is seen in split brain patients.

[mraptor]

@adam

So maybe the cortex is doing the decision making at column level and the basal ganglia / other old brain parts are deciding which output of the neocortex makes most sense (global level). This way you avoid to search for a central decision maker which I think more closely captures the behavior which is seen in split brain patients.

So you are saying … CC connect to another CC (in the cortex) via thalamus to generate the Actions (RL&Planning) according to the model, which then are a sent to BG to be filtered.

My interpretation : Cortex generate Virtual Motor Actions according to model held in Cortex which are then filtered by BG and trigger Physical Motor programs that move muscles

[adam]

Yes exactly.

From a macroscopic level it seems pretty obvious to me that it has to work like that. However the implementation details on the level of single CC and how they connect to other CC… probably we have to wait for Numenta to resolve the mystery.

Btw: BG is known to interconnect with Cortex and is speculated to implement this gating / filtering: this can even be found on Wikipedia and seems to be more accepted knowledge:

Popular theories implicate the basal ganglia primarily in action selection – in helping to decide which of several possible behaviors to execute at any given time. In more specific terms, the basal ganglia’s primary function is likely to control and regulate activities of the motor and premotor cortical areas so that voluntary movements can be performed smoothly.