I am still reading Jeff’s new book but I will offer my current view of subcortical structures and edit if I see any differences after reading the book.
I see that the subcortical structures are like “training wheels” to kickstart the learning programs in the cortex. In the beginning the subcortex does the heavy lifting of running the body. As the cortex becomes more capable it slowly takes over the old brain and overrides it on most things. As has been pointed out - you can only force yourself to hold your breath so long; eventually the old brain fail-safe mechanisms reassert control and take over.
You speak of reinforcement learning. I don’t think that is as easy as “dopamine receptors drive reinforcement” learning.
Take the amygdala - it comes preprogrammed with visual primitives, and perhaps other sensory primitives. It is heavily interconnected with the hippocampus - the digested version of the world at the top of the WHAT/WHERE streams. The information has received as much analysis as the cortex is capable of performing on your experience.
I say that a key feature of the hippocampus is to buffer episodic experience. You can’t know if an experience is good or bad until it is over. The amygdala is matching up the stored primitives and sending back good/bad judgments to mix in with the current episode.
In sleep, this mix of episode and good/bad judgment is pushed back onto the cortex by the process we call dreaming.
This process happens to every experience throughout your life and builds judgment of good or bad onto the object particles in the cortical stores. All of them.
You generally don’t reason out every situation from first principles - you use the building blocks of the “best flavored” sensed and remembered objects to formulate a plan of action in the ventromedial prefrontal cortex.
The good/bad features in the amygdala are somehow updated as you grow older; this is one of the areas I have been studying.
In regards to how that fits with cortex that is divorced from the fearful old brain - humans with damaged amygdalas have very poor judgment. I have trouble envisioning a useful intelligence that does not have the same basics drives as a human every forming empathy or a value system that we humans would be comfortable with.