The evolutionary history of integrated perception, cognition, and action

I don’t know about you guys, but I had a fair amount of preconceptions blown away since I read that paper from Paul Cisek… originally put forward by @Bitking again:

I’m currently trying to read more from that guy… Just stumbled upon a filmed intervention following at least the intro of that paper, and going into some more details about our current organization, for the video-inclined.

Now I had to challenge some of what I knew… and even large parts of most global cognitive models on those grounds… I have a hard time trying to re-interpret all the puzzles pieces towards a more global understanding. Some of the very questions we ask ourselves when classically thinking about the brain need to be turned on their heads.

If some of you wish to explore this direction we could do that here.


I am definitely interested in this topic. I also think it is a key insight to understand the functions of the different brain structures.

I’ll read the paper of Paul Cisek, it looks very interesting.

Even if it is less technical, you may be interested in Joseph LeDoux last book “The Deep History of Ourselves”:


I ain’t an old-timer here. And it’s been a relatively short time since I began to study neurology. But from the onset I’ve been interested in some comparative studies and the evolutionary side of the matter.

What’s at stake here however is not simply to get a little deeper in our understanding, using evo studies… it is clearly rethinking a number of fundamental assumptions about cognitive models.

That author proposes that the usual division of concerns : Perception-Cognition-Action subtending most of our thoughts about brain functionality is not a valid cut after all. And that by studying the evo, we may get a far better insight on the “good” questions of whys and whats and hows.

In this paper, you’d find lots of references to researchers with interesting… kinda… ahem… “subversive viewpoints”… on a lot of how we come to approach neurology itself.
Mind you, I asked myself if I was attracted to them by the very fact that they challenge common assumptions, and that it caters to my internal hooligan… but I’m pretty sure they do have a point in the end.

A few example thoughts, maybe? to illustrate what I’m after…

Brain theorists often discuss what has come to be known as the “binding problem”.
Neurophysiological experiments have shown that different, and often very distant brain systems specialize in different aspects of perceptual analysis and behavioral guidance. Given such divergence of information, how is this information ultimately integrated into a coherent unambiguous whole?
Many different kinds of binding problems appear to exist. These may be classified into “within-system” binding and “cross-system” binding.
It is certainly plausible that distributed representations in the cerebral cortex are unified through a general mechanism for solving the binding problem. Perhaps this mechanism involves temporal synchrony, perhaps it involves complicated anatomical connections, perhaps it’s something else altogether. However, it is also possible that different kinds of binding problems are resolved with different mechanisms, each dependent upon the requirements of the behavioral task at hand. Furthermore, it is possible that some of these behavioral tasks do not require independent representations to be bound at all.

Paul Cisek in “Binding Through the Fovea: A Tale of Perception in the Service of Action”

Seeing does not require compensating for the effects produced by eye shifts in order to ensure accurate accumulation of partial views into a composite patchwork projected on some internal screen. There is no need to recreate another world inside the head in order for it to be seen.

J. Kevin O’Regan in “A sensorimotor account of vision and visual consciousness”

“Everyone knows what attention is” (James, 1890) is one of the most popular quotes from William James and certainly the most famous statement about human attention. We argue, however, that the overuse and popularity of this statement in cognitive research has been detrimental to progress – that in fact, no one knows what attention is. More specifically, we argue that the concept of “attention” is one of the most misleading and misused terms in the cognitive sciences. In the present paper, we stake the position that the term “attention” should be abandoned and the nature of the research in this area be re-conceptualized to focus on the subsets of processes and mechanisms that lead to task-specific performance.

Bernhard Hommel in “No one knows what attention is // Time for action: Reaching for a better understanding of the dynamics of cognition”

However, the neural coding metaphor is used more broadly, in ways that are less obviously related to communication problems. For example, neurons in the primary visual cortex encode the orientation of bars in their firing rate; neurons in the auditory brainstem encode the spatial position of sounds, and neurons in the hippocampus encode the animal’s location. The coding metaphor applies to these situations in the sense that there is a correspondence between neural activity and some measurable property. In this sense, the metaphor applies equally well to any situation where two measurable properties co-vary. But as the linguists Lakoff and Johnson (2008) have argued, the metaphors that pervade our language are not neutral; on the contrary, they form the architecture of our conceptual system. For example, seeing the heart as a pump is quite different from seeing it as an information processing device encoding walking speed in its beat rate, two equally applicable metaphors. Is perception a communication problem? And if so, with whom?

Romain Brette in “Is coding a relevant metaphor for the brain?”

the task of mapping the concepts of Fig. 1 to neural data has proven difficult. The neural correlates of putatively unified functions, such as “working memory” or “decision making”, appear to be distributed throughout the brain, whereas other concepts that one might expect to be separate, such as “attention” and “intention”, are often intermixed in single regions, sometimes even at the level of single cells (Cisek & Kalaska, 2010). The bridge between psychological concepts and neurophysiological mechanisms is difficult to establish, leading to proposals that many of the questions being asked are perhaps not ideally framed

Paul Cisek in “Resynthesizing behavior through phylogenetic refinement”


Some older Cisek video, where he’ll present some of the “control loop” ideas to a more AI-oriented audience.

Review of the history of some ideas in the field of brain theory is interesting. Then he’s proposing the two fundamental control loops available to life and all animals and go straight to developping this idea for vision in the wild.

I don’t know if that opinion needs to be stressed here, maybe this would be beating dead horses for some, that there are multiple pathways, and to be reminded of subcortical structures. But still there’s a shift in viewpoint imho, however slight, to depart from the very information-oriented models we usually consider.
At some point he’d mention deciding between conflicting actions and I’ve been thinking… those bitkingy grids could match.
Nothing too evo there yet. Last question in the vid still decided me to post it anyway: interestingly he’d propose adding “meaning” as an afterthought to action.

I’ll try to find the evo slides he used in the OP vid and add a few comments around them, so that we can discuss that part even if you’re not thrilled by the control-loop ideas.

1 Like

My biggest concern with the otherwise excellent Cisek model through following the evolutionary trail is his blindness to the important role that chemical signals through the olfactory system provides and its possible role as a co-factor in the development of the hippocampus memory system. This places some form of personal experience very early in the evolutionary process and has co-developed along with the rest of the brain hardware.

His general outline matches up quite nicely with my understanding of the relationship between the lizard brain and the cortex.

The decisions that he alluded to are informed by the fancy cortex but remain in the firm control of the subcortical structures. As he mentioned, it has been extensively documented that the cortex (and the general mechanism of experience) becomes aware of decisions well after they have been made.


Can’t find those slides or illustrations easily. Most seem to revolve around the work of Luis Puelles or some shared efforts around him (and his so called ‘prosomeric’ model), if you want to dig those with me.

Could this be the slide deck you are looking for?

A useful search trick, google "paul cisek slides’

You can put all kinds of interesting things in your “slides” search.
Oh, and you may like this slide set too!

I have been reading Stephen Grossberg for years - he has many interesting ideas.

1 Like

I swear I did !

Unfortunately no, I’m trying to find the other one… from “The evolutionary history of integrated perception, cognition, and action” video. There were slides showing nice colored successive steps along our line, with hypothalamus, evo of pallium, etc.

Does this help?
Google: paul cisek telencephalic foraging system
Show as images.
Many similar, but not exact hits.

Also - affordances …


I think that Paul Cisek would be an excellent choice for your “interview with a neuroscientist” series.


We’ve reached to P. Cisek and may be able to get some illustrations after all.

In the meantime, I’ve started to read the most excellent book “The Evolution of Memory Systems, Ancestors, Anatomy, and Adaptations” by Murray, Wise and Graham.

Really a very nice read so far, and I’d advise anyone to have a look.

I’m frustrated by a missing puzzle piece for my understanding, however.
In their description of an evolution line, they’d present (with a terminology I won’t embrace yet if you didn’t read the book) some basic universal capacity (as in, almost any-animal-with-NN across all lineages) as a first component of learning (or memory).
Somehow they tie this first component to pavlovian findings. I’m not disputing the claim, however for it to be possible at all, in some of the examples cited, it seems to me that there should be some form of… “retainment” (don’t want to use “memory” there yet) of a sensory condition, long enough to tie it to a “future” (how long, btw?) valued outcome.
Some STM, if you wish.

Being here on this forum, next best thing beyond such conceptual knowledge is… I’m mostly interested in how to model it, so it’s quite important to me that we’d be able to describe such retainment system in very very basal - and basic - NNs.
Yet the only pavlovian experiment on a “small and simple” being + neurological result I found so far seems to involve the wiring of a sensory input with a (sadly) concomitant internal representation:

“Sadly” cuz, whatever the internal representation, I’d have no problem imagining wiring it to “current sensory input”.

Wiring it to some “past (even recent past) sensory input” is what eludes me.

Any idea on this, anyone ?
Recall we’re speaking here of very primitive NNs, so if that idea could just… not-include HC, that would be neat.


Firstly, thank you for originally posting Cisek’s work to this forum. I have consumed all of it that I could find and it really resonates with me.

In my reading of Cisek’s paper “Resynthesizing behavior through phylogenetic refinement” he does seem (to me) to emphasize the role of the olfactory system in the development of the hippocampus. But perhaps that’s just me reading too much between the lines (?) He does state in the paper that “the present article will merely present a basic sketch”, which he intends to build on over the years. I look forward to future papers from him elaborating these ideas.

1 Like

I’ve found some additional related material y’all might be interested in.
First, here are 2 more talks and an interview on YouTube:

In that last one he mentions he’s writing a book! I look forward to that.

Finally, the “Unsupervised Thinking” podcast discussed his most recent paper, linked in the previous post: (Episode 48)

In private communication with Doctor Cisek he mentions the important role of the olfactory system but felt that with limited time in a talk that bringing this in would take too long.

He intends to cover the olfactory system in much greater detail in the book he is working on now.

1 Like

Excellent! It is very interesting to me to think about early terrestrial animals facing new navigational challenges and how their brains responded evolutionary. Perhaps we can thank a nocturnal ancestor for some of our neural innovations. I wonder if the “long night” faced by the surviving mammals after the KT extinction event played a significant role.

That period likely played its part, but I’d cast some doubts about it being night vision. Current understanding seems to indicate that we mammals had been nocturnal for quite some time, and that diurnality only returned to our line with anthropoids (new&old world monkeys, apes, and us)

Thanks for the additional information, very interesting. For clarification, rather than “night vision”, I was referring to the development of brain structures for navigation that aren’t dependent at all on vision but instead can make use of non-visual senses such as olfaction, whiskers, etc. It seems to me that if you can’t see (much) you are more reliant on the ability to build internal models of the environment to facilitate navigation. Hence the link highlighted by @Bitking between olfaction and the hippocampus.

1 Like

I’d need to read some material again to be sure, but iirc it is thought there were also large enhancements to those auditory, olfactory, and somato- (eg. whiskers) sensory pathways at the time of the mammalian ancestor already (long before KT).

That being said, involvement of the HC with olfactory pathway is likely antediluvian compared to that. @Bitking calls for a model taking these old loops into account. That would be sometime towards the vertebrate or chordate split. (fishes, anyone? ^^)

1 Like

If this approach interests you here are some additional papers from researchers that were mentioned by Dr. Cisek:

New perspectives on the hippocampus and memory

Zounds! $90!

I think that from the general topic you will find these interesting:

So many papers, so little time!


Wow, thanks for all that! So little time indeed… I guess now I know what I’ll be reading for the next month :wink:

1 Like