Actually, I think his affordances are more similar to the behaviors that Jeff talks about. If I’m understanding the paper correctly, his affordances seem to be the set of potential behaviors associated with known objects that are currently or potentially accesible to the agent at any given moment.
Affordances, behaviors, objects, movements, actions, objectives, tasks, goals, etc. All represented in the same fabric using a homogenous mechanism. This circuit we’re trying to define, depending on how it is used, is a memory storage / retrieval device for any of these things.
Maybe the difference is where in the brain the memory is pooling. The Cisek work suggests affordances pool in the PPC area in the dorsal stream. Our work suggests object pooling also occurs, and I’d go as far as saying the allocentric representations we always talk about are probably in the ventral stream, likely simultaneously.
Not looking to start a long drawn out fight here but there is more than one way to look at an object.
We are used to thinking of an object as some pattern or sequence in a map that reliably signals some object - usually some perceived object.
When you think of a behaviour you have to ask “how is that stored in the cortex.” Again, it is likely to be some pattern or sequence. This is about the same thing as an object; cortex is pretty agnostic in what is stored in it.
I see an affordance as the union of the cluster of features that make up your perception and the cluster of features for behavior that happens to be the best fit to your perception. An affordance.
I think of it as the senses priming the selections of possible behavior implementations, with some drive coming out of the subcortical structures as a strong template for these behaviors.
This is all very high-concept at this point so what does it look like in practice?
Perception -> what/where stream -> Association hub & temporal lobe -> subcortical structures -> lower forebrain -> unfolding into some action -> motor drivers -> action!
The spatial extents are hard to put into words so maybe these drawings will help. As it is these are pretty busy - if I put this all in a single drawing it would be incomprehensible.
These are the parts I will be describing below. I broke out the sub-cortical structures for clarity in this structure key. In the following illustrations, the sequence is color coded to match the descriptive text below the picture.
This last bit is the priming of action plan selection based on what is being perceived. This is also includes some lobe-to-lobe connections.
In this sequence the sensory stream has object features that are collected in the association regions that communicate with both the temporal lobe and the frontal lobe. These object are composed of features that are stored with built in judgement of positive and negative associations during exploration and learning so that in planning these object stores contribute to action selection. The “objects” stored in the different regions of the brain are vastly different but the common lingua franca of hex-grid coding allows inter-communications.
It’s worth adding that while this is decomposed into stages to make it explainable, this all happens in a continuous stream and could be considered as a purely parallel process.
The pragmatic representations on the dorsal stream may decompose a same stimulus in more than one potential action, on different parts of the cortical maps. And each may have its own metrics. The “what” pathway is likely receptacle to several other kinds of representations.
You know that psychometric experiment (and famous illusion) where a circle is surrounded by other circles… depending on their relative sizes, you’d describe the central one as smaller or greater than it really is?
Well… if instead of describing it, you were asked to reach and grab for that central circle, the gap between your fingers would be the correct one. Some of them representations are this-illusion-proof. Hence not the same. And probably not even using same referentials.
It’s clear to me that Cisek builds a case based on sound experiments, and discards misconceptions based on old and vague assumpsions. But it lacks the concrete mechanics and models that for instance Numenta’s spacial pooler or displacement cell models descibe very precisely and specifically.
Of course I have to read more of his papers and watch more of his presentations. But right now it looks interesting but not totally convincing.
I’ve been in an email conversation with Paul Cisek, and in response to my confusion about what “representation” means do different people, he said this (and allowed me to post it here):
The business of “representations” is a bit of a semantic mess, because different people mean different things. Some just mean that neural activity correlates with something, which means we can decode it, and by that definition they obviously exist. However, by that definition, heartbeat represents running speed… so it’s a bit too inclusive. On the other extreme, some people believe that in order for any behavior function to take place, the external world must be internally “re-presented” inside the brain. However, this then raises the question of who looks at that representation (a homunculus?). You’ve probably heard all this before…
Anyway, my take on it is to think of representations along a continuum: At one end, we have simple feedback control circuits, in which internal states correlate with external stuff only insofar as being dynamically coupled to each other. I think a lot of neural activity is of this type, from the firing of cells in the stretch reflex to the firing of cells in the premotor cortex that guide the hand to a target. All of these representations are only understandable in the context of the circuit within which they reside, and the functional role of that whole circuit. For example, since premotor cell activity also varies with things like arousal, number of targets, target value, etc., you cannot really “decode” anything from them unless you know all of those variables (e.g. are running a highly controlled neuroscience experiment). I think the vast majority of neural activity is of this type – dynamical coupling that drives the system to a good state.
However, some neural activities are less modulated by internal states, and more “objectively” related to the outside world. For example, while cells in the insula might respond to an apple only when you’re hungry, because they play a pragmatic role in guiding feeding behavior, cells in IT cortex might respond to apples regardless of hunger or any other aspect of context. These are what I would call “descriptive” – in the sense that their functional role really is to convey information about the outside world. Frankly, I think such things are pretty rare in the brain. But the value of thinking about representations along such a continuum is that we can think of evolutionary scenarios in which a pragmatic representations, embedded within some control circuit, could have gradually become “divorced” from all of those contextual influences – for the simple reason that sometimes it’s good to have objective knowledge. For example, if I’m wondering around and see an apple when I’m not hungry, it’s useful to make a note in my cognitive map so that I can come back later when I am hungry. Again, these kinds of representations are probably a small subset of neural activity, but they probably do exist and have a phylogenetic history.
Um, if I understand what is happening, the older structures of the brain use a simplified version of the world, predigested by the cortex to make it understandable to these brain structures. Likewise, the commands from these older structures are relatively simple and the cortex elaborates those commands into more sophisticated actions.
I think this view is better explained in the context of distributed semantics. A fast heartbeat is certainly one component of a concept like “running fast”
Is this question answered, though, just because we say that most activity is context-specific rather than “objective”/“descriptive”?
This is from a paper exploring the quasi philosophical questions of whether we should think of neuronal rates or phases as “coding”, at all.
Probably not, but I see why the philosophical question is, in fact, welcome.
What’s at stake for us, students of brains and wannabe modellers, is how we may get inspired in finding the pertinent questions, to get to insightful answers.
So I don’t see the true nature of the debate as ‘are brains representing such and such’? Rather, as an evaluation of the benefits of recentering our view on real (and sometimes initial) purposes of nerve cells.
We too often take the view that processing of sensory stimulus shall ultimately lead to a representation allowing us to “make sense of the world”… and we refine models so that they encode things we understand, and hopefully show stepping stones to those representations, imbuing the world with “meaning”. While, as P. Cisek would put it, what they do is often more ‘pragmatic’ than that. What sensory neurons activation was meant to carry originally was often the control of a quite straightforward behaviour.
Which spawns 2 thoughts:
A pragmatic representation is not necessarily readily amenable to understanding. As the author of the ‘Avida’ alife software said somewhere, optimized representations (such as the ones found by evo search, increasing a behaviour’s fitness) are as cryptic as compressed files… if they were not, ie if they were understandable, they wouldn’t be optimum.
Although… we “do” make sense of the world. And there shall exist something we could relate to somewhere. And some recognizable ‘encodings’ at times. Such as face cells and the like. But they’re likely high in the hierarchy or emerge from the concurrent activation of lots of more basic loops, having found some new purpose together, and possibly after having reached some critical “mass” (or we easily forget we’re so chauvinistic about human big brains and intelligence or consciousness at times).
That being said, those are my takes only. And I don’t understand in the slightest how plastic nets with STDP cells would finally decide to wire “usefully”, be it to specific motor nerve cells or to drive seemingly hard-fixed purpose neurons such as grid or orientation cells… But I guess some of you people are quite knowledgeable on these questions.
I don’t believe there is any reason to expect that some outside observer should be able to “decrypt” a brain’s activity. But does that imply there is nothing “encoded” in the first place? We could not possibly manipulate the world to the degree that we do if our brains didn’t encode vast amounts of information about it.
Of course, I get that Paul Cisek is referring to a far wider scope of brain architectures when he refers to so-called objective neural activity as “pretty rare”. But I think most of us on this forum are interested in mammal brains from the perspective of the cortex, where such activity seems (at least to me) as kind of its whole purpose.
I also see these as the same thing. Behaviors and their effect on the relationship between components of a composite object can themselves be components of other composite objects.
Definitely. The idea of the cortex modeling the world in a way that is primed for action selection is pretty compelling.
Our cortex is a huge part of the full “scope of [our] brain architecture”. So if you’re ready to accept something as pretty rare overall, I don’t think its rarity can be discarded as “not applying to cortex”.
TBH, I don’t know about the “pretty rare” thing myself. But I’m becoming more and more aware of the fact that cortex didn’t just appear out of the blue. If practical representations were indeed a thing, then even cortex deals with a whole bunch of those, and I’d bet still today.
Most of our cortical areas are from great apes, we have a great deal of those homologues still with primates, a fair amount probably homologous with mammalian LCO, and some allegedly before that. Start of our visual “streams” in particular are already wired to the pallium of lots of vertebrates. And even our hippocampus friend sitting “at the top”(?) has a very long history, before we knew how to breathe air.
