If you try to make cortex to the RL then yes, I see how there is no good answer.
Cortex learning of transitions is very local learning and knows little or nothing about goals.
It learns to model the world and signals when something novel is happening.
If my understanding is correct, you are correct, the “older” subcortical sections of the brain drive states and goals, and reinforcement learning.