Semantic Memory Mapping in the Neocortex (Visual and language memory co-locate with transitions)

This great article reveals interesting new insights from J. Gallant’s lab (UC Berkeley) into the semantic mapping taking place in the neocortex, in which surprisingly, visual memory locations (fMRI voxels) transition to their language equivalents. There appears to be a semantic order that goes beyond the type of perceptive input in which visual and language memory co-locate). This research reveals an over-arching semantic organization in given portions of the neocortex.

I just posted it: Generalization from images to words mapped in a brain region - #2 by EEProf :slight_smile:

1 Like

Sorry, I did not realize you also just posted it. Great topic.

Can we join the threads? Or can this be moved to the other thread?

I would be very interested in opinions about this finding, with respect to the TBT architecture. Considering that these semantic regions contain multiple so called voxels (fMRI mini regions) which are very large (about the size of peas), this indicates that multiple TBT regions are most likely also contained within a semantic region. This, in my opinion, is fully compatible with many of the TBT premises and findings, but perhaps gives us an additional new dimension to consider. In TBT it was already assumed that some TBT regions are within given perceptual areas, which would make them specialist for that type of sensory input. With this new knowledge, we now also have the new possibility to consider that these TBT regions are more importantly semantically specialized. Semantic specialization seems intuitively to be more significant than the perceptual modal forms (i.e. sensory inputs or sensory sources). A “voting” process for consensus building amongst semantic specialists seems more behaviorally valuable than just amongst sensory specialists. Does this intuitive hypothesis resonate with anyone here?

Cue the theme from “WestWorld”:

“Subjective conscious mind is an analog of what is called the real world. It is built up with a vocabulary or lexical field whose terms are all metaphors or analogs of behavior in the physical world.”
–Julian Jaynes in The Origin of Consciousness in the Breakdown of the Bicameral Mind , 1976.

Doesn’t matter, to me it’s just more evidence of hierarchical nature of the cortex, which Numenta has been de-emphasizing lately. Some folks here have been skeptical about it too.

I also have never abandoned the concept of hierarchy as an organizational principle in the brain. Perhaps it is not a physically hierarchically structured wiring, but at some levels even our HTM (Hierarchical Temporal Memory) name has always been a central element, which is a building block of the TBT concept as well. And from a more abstract perspective (less physical) hierarchy most certainly plays a fundamental role.

But what we have to always remember is that some of these terms, like hierarchy, semantic domains, functional domains, are emergent phenomena. And as such they can coexist at different levels. One organizational concept does not strictly exclude the other. And if they coexist, then they most certainly enhance each other. (I say if they coexist, then they are complementary. Why? Because one lesson of evolution is that nothing intricate is there for no reason).

Yes, I take this elaborate formulation to simply state that our brain generates and maintains a representation of the real world. And a representation is like a metaphor, or an analogous model of reality in Julian Jaynes vocabulary.

1 Like

This point is important. We already know that sensory and motor functions co-locate (meaning they are physically close to each other). And we also have known that sensory-memory integration (that is perceptual and memory) is also taking place in the cortex. So, summarizing we have a sensory-memory-motor integration (co-location, sharing some of the same neurons). And now we can observe a big picture with semantic regions which integrate these sensory-memory-motor) mini-workflows.

1 Like

I find it making perfect sense. Actually I cannot imagine otherwise. Modal perceptions (from different sensory inputs) for every concept/object have to converge some where in the cortex (e.g. concept “green” upon seeing green vs hearing the word green, separately or simultaneously), for our mind to maintain a coherent/consistent/persistent perception of the external (and/or internal) world. How else can it be done?


In some sense, everything beyond voxels is “emergent” :). But hierarchy represents compression, and lossless component of compression is a measure of GI. So, from functional perspective it’s a prior, not emergent. As such, hierarchy exists on all levels of a brain: from single neuron (many inputs to single output) to whole-system (task-positive network to default-mode network).

1 Like

There is a physical hierarchy, for example L5 → thalamus → L4 only goes up the hierarchy.

That doesn’t mean hierarchical feature extraction. They talked about it in some research meeting videos. Something about references (in the programming sense of the word). It might be like a hierarchy of attention, I forget.


I fully agree. As mentioned, it makes sense that sensory inputs converge and end up in some sort of sensory fusion process. But the main thought I was trying to express here, is that if we think of the TBT “voting process” (which is actually more of a consensus building process), this new view of the cortex as broken down into semantical areas, instead of sensory areas also has some important implications about how the TBT consensus building takes place. Some of the TBT mini-regions participating in the voting, could for example come from a semantic area for houses and another one would possibly come from a region for animals and yet another would come from an area for vehicles. So the constituents which are voting have semantic perspectives instead of sensorial perspectives. (Each TBT mini-regions has models of the world for that semantic domain). This would mean that new sensory inputs are being submitted to a semantical competition. In formulating the TBT Jeff already assumes semantical competition, but this new evidence about an underlying semantical organization of the neocortex strengthens this view.

Prior to this new evidence for semantic memory mapping, we conjectured in TBT that a semantic identification of objects occurs across many TBT regions and the most active ones win the voting process. This still holds true with this new revelation. But now it becomes more clear that the location of the TBT mini-regions also correlates with certain semantic models which locally predominate in a given region.

Here is the primary source for the article:

Visual and linguistic semantic representations are aligned at the border of human visual cortex | Nature Neuroscience

Popham, S.F., Huth, A.G., Bilenko, N.Y. et al. 2021


Semantic information in the human brain is organized into multiple networks, but the fine-grain relationships between them are poorly understood. In this study, we compared semantic maps obtained from two functional magnetic resonance imaging experiments in the same participants: one that used silent movies as stimuli and another that used narrative stories. Movies evoked activity from a network of modality-specific, semantically selective areas in visual cortex. Stories evoked activity from another network of semantically selective areas immediately anterior to visual cortex. Remarkably, the pattern of semantic selectivity in these two distinct networks corresponded along the boundary of visual cortex: for visual categories represented posterior to the boundary, the same categories were represented linguistically on the anterior side. These results suggest that these two networks are smoothly joined to form one contiguous map.


Evidence is mounting on the existence of a visual-semantic mechanism in the occipital region of the neocortex. An important unanswered question is whether these semantic structures act as primitives which are used and invoked by other episodic (temporally) stored and recalled memories. Does the brain have both a visual-semantic occipital macro-region and an episodic-temporal (frontal) macro-region of the cortex?

If so, then language would emerge from the interactions of both. (That would be my intuitive guess). And the temporal lobe, where much of our linguistic activity has been correlated to, is indeed located in these gaps between the other two macro regions.

1 Like