Numenta Research Meeting - September 23, 2020

Science Current Research
1

Jeff Hawkins brainstorms some ideas on minicolumns, in a continuation of a recent concept he presented in Numenta’s July 27, 2020 research meeting. In the previous meeting, he hypothesized that minicolumns represent movement vectors. In this research meeting, Jeff suggests a new mechanism for calculating reference frame transformation that ties into his minicolumn hypothesis. He suggests that the movement vectors of the minicolumns’ upper layers are allocentric, while that of the lower layers are ego-centric.

July 27, 2020 Research Meeting:

2 Likes
2

@jhawkins and the team discuss how two types of sensory input (static and flow/change) are processed by two parts of the minicolumn, and Jeff explains that there is evidence for it in the visual cortex and suggests that this happens everywhere in the brain.

A question that comes up is that for this to work throughout the cortex, would mean that minicolumns must have evolved from two sides at the same time. One to process two different subtypes of input, and one to connect those two types.

So to illustrate this, it would be like evolving to process visual input from two visual streams, then be copied to another part of the brain where lets say tactile information enters the brain, and then finding two equivalent subtypes of that tactile input and connecting the right type to the right partition. And then doing that again for hearing and presumably also for other more abstract types of processing in the neocortex.

I’m not doubting it, I actually find it fascinating, but how would that work? How likely is it that the correct subtype of input gets connected to the correct partition in the minicolumn in a completly different working environment?

2 Likes
3

If you think of it as “a basket of features” in the association areas this starts to make sense.
My Hex grid thing has a pattern of sub-nodes that can consist of any feature that is connected to any other in the overall activation pattern.
So for a given macro-column, the inputs from different sensory modality can be part of any collection of features for some object.

2 Likes
4

With modalities, do you mean the difference between static and change of a particular input stream?

5

No - I am speaking of different sensory streams converging on the association regions.
The hierarchy acts to extract micro-features of temporal and spatial extent, projecting all of these to the central region of the parietal lobe. This makes available all of the sensory features in a rich soup of spatial/temporal micro-features that can be associated with the hex-grid (or TBT if you think of it that way) to signal a recognized object to the other lobes. Each macro-column in this area has some portion of this soup of extracted features feeding to the dendrites to form SDRs formed of conjunctions of these micro-features. The individual macro-columns act locally but the lateral connections act to form a global recognition.

1 Like
6

Another thought, concerning the transform between the two minicolumn partitions, would it be possible to make a simple animation of how the processing of a very simple imput stream would look like? I understand we don’t know how this processing happens, but what would you idealy like it to be?

(I wish Matt was still with us. I’m sure he could make a great demo clip showing something like that).

So lets take an example of a dot moving in a straight line towards us but not completely into our vision axis. The dot moves from slightly above our horizon to slightly below, at the same time from slightly left to slightly right, and also from 2 meter away from us to 1 meter in front of us. Naturally, neither our head nor eye(s) move while this happens.

So I suppose there is one particular cell on our retina that will trigger only when the dot moves across it, and a somewhat larger array of cells that will fire for this particular change vector (top left to bottom right).

Sub question: I’m not even sure if it would matter if the dot moves towards us or moves away from us. Maybe that motion is inferred at another level in the hierarchy?

But for now, what would an ideal vector representation look like for the static input, and for the flow input?

The static (I guess) is simple. It would be a binary memory representing the one spot on the retina, and it would only be active when the dot moving towards us crosses that spot. Is this right?

Sub question 2: would a minicolumn be linked to only one retina spot, or would it make more sense to have several spots connected to the minicolumn. And if several, should they be adjacent spots or spread out over the retina? Would they all be inside the same flow receptive field?

The flow vector I have more trouble imagining. Since the receptive field is larger than for the static input, I guess it would be an area around the spot connected to the static input. Maybe circular, or elliptical? But when the dot moves across it, it would generate a non-binary value that would range between maximum if it represents flow in the exact axis the dot moves in, and zero if it moves orthogonally to it. Am I right to assume that there would be only one flow signal connected to our minicolumn?

And how would that flow value be represented? Is it a binary array, with more bits active the more the dot movement matches the axis our minicolumn represents. Or is it a single flashing bit where a higher frequency represents the matching axis?

Sorry if these are noob questions, but I think we should be clear what we are talking about. Often the biggest problem is communication. In my experience especially among engineers.

2 Likes
7

At 45:11 @Max_Bennett mentions a blog post. Could you please link?

And I have @Falco’s questions too :smiley:

1 Like
8

Sorry, it was Niels Leadholm that mentions the blog post.

9

I found this:

3 Likes
10

I would like to respond to the idea, that every cortical column deals with Flow & Static input. Since Jeff mentioned it a few month ago, I was fascinated by this idea. The observation that we can build a map of the environment, just by watching a video or playing a computer game is awesome. There is no doubt that the cortex needs to be able to do this solely based on sensory information.

However I was kind of skeptical that this applies to all sensory modalities. What made me doubtful? If this is a property of mini-columns / columns, this would imply that the full cortical sheet would always ( or better “neuroscience”-always = mostly :wink: ) have two inputs. I think that if it was correct, then there should be much more evidence, like having two different thalamic laminae (similar to the LGN) for every sensory modality and for association thalamic nuclei. However this seems not to be the case.

Then last week I read the following (famous) paper:
(Why famous? You probably all know this image: http://vis.berkeley.edu/courses/cs294-10-sp10/wiki/images/6/6b/Felleman_CC_1991.png)

In the section “Intertwined Processing Streams in the Visual Cortex” they describe very accurate, that already in V1 the two processing streams kept separated, at least to some degree. They even assign distinct streams of processing to the separation seen in the thalamus. Basically the two-stream hypothesis (https://en.wikipedia.org/wiki/Two-streams_hypothesis) is emerging from there. And they are able to trace this back to the thalamic layers of the LGN.

  • Magnocellular (Jeff: “flow” / movement):
    • L4B & L4C_alpha in V1
    • thick stripes V2
    • other areas: V3, MT, MST
  • Parvocellular (Jeff: “static” / features):
    • L4A & L4C_beta in V1 & superficial layers of V1
    • thin stripes V2
    • other areas: V4

For me this is evidence enough, that this theory has issues and needs to be adjusted, somehow. However as the paper is rather old (30 years!!), there might be newer data available, that supports the theory.
I would be happy to read your comments!

1 Like
11

This is tricky because the two types of cells could be organized in other ways than thalamic laminae, like subnuclei, a matrix, two separate nuclei, or just mixed together. It can be hard to judge this because the thalamus is has intricate 3d structures.

That might be right but it’s hard to tell. Do thin / thick stripes in V2 have separate minicolumns? It’s possible they’re just different sublayers being thicker or thinner in a stripped pattern. Sublayers can probably be 3-d rather than completely flat sheets.

2 Likes
12

It is true that the projection might be not so simple, and a clustering of those cells into flow/static was not done so far for other sensory modalities, because we did not observe structure in the thalamus.
However I would argue, that if it is the case, that we can see a very structured LGN (magnocellular and parvocellular input separated into layers) and we have other sensory modalities, where such structure unknown, then it seems more probable to me, that the layers of the LGN are a unique feature of the visual system, rather than a general rule, how cortex and thalamus interact.

I know only one general classification system of cells which exists throughout the thalamus: This is the matrix/core system. However both magnocellular and parvocellar cells in the thalamus are of core-type. So the matrix neurons must do something else.

I am rather new to the thalamus and its structure. Do you know an example for a different sensory modality, where classification similar to magocellular/parvocellular layers is possible?

V2 thick / thin stripes: Those are macroscopic structures, visible to the eye, and as such different sub-areas / minicolumns within V2. (For reference: http://vision.ucsf.edu/hortonlab/Research_Mapping.html)

13

It’s possible in the rat/mouse whisker system. The thalamic cells haven’t been studied so much and are pretty sensitive to anaesthesia so it’s unsure. Some have larger receptive fields and are less responsive, which could mean they require a bunch of whiskers being deflected at the same time. Some are responsive to fewer whiskers, which could mean they detect features rather than flow.

In whisker thalamus, we know the structure to some degree. I’ll go off my memory. The main thalamic nucleus is VPM (there are others, mainly POm, but seems more like the pulvinar than like LGN). For each whisker, it has a long cylinder-ish structure. In the middle is called the core (just the name, not a reference to core/matrix cells). It receives sensory input from one or few whiskers (via sensory trigeminal nuclei) and sends axons to a corresponding barrel in the primary somatosensory cortex. The two ends of the cylinders are the tail and the head, which receive input from other sensory trigeminal nuclei which relay info from more whiskers. The tail and head of VPM project to S2 and larger swaths of barrel cortex, in the spaces between the barrels.

Yeah, then they have separate minicolumns.

Take M1 as an example. It has very thick lower layers, but it still has L2/3 and L4. Perhaps M1 is a specialization to emphasize egocentric processing but still does allocentric processing too.
It’s possible the thin / thick stripes in V2 are the same. Just because they seem to handle magnocellular or parvocellular doesn’t mean that’s all they handle.