Jeff Hawkins explains how introspection can be a helpful tool in neuroscience research. He first gives an overview of what a column in the neocortex needs to know to recognize an object. To recognize an object, a column must simultaneously infer object, location, orientation and scale. The team then extensively discusses how scaling works in the columns. Lastly, Jeff proposes four possible explanations for how the neocortex can represent objects with different orientations.
It seems like the egocentric object + location + orientation representation would have some similarities to the egocentric room + location + orientation representation. There could be a set of property/place/orientation representations and a set of grid-cell like transformations that would allow any such representation associated with a specific object to transition from one state (location + orientation) to another. One would expect that there would be sufficient overlap in the SDRs for us to be able to recognize (or conjecture) the persistence of the object despite the changes in itâs location and orientation.
Concerning whether weird deformations or stretchings @subutai and @jhawkins discuss at 27 min 36 s are recognized as same objects, how about watching an object across a transparent bottle filled with water, or an irregular magnifying glass, or an object lying on the bottom of a filled sink? Donât we recognize that object? Itâs litterally the same object, and it doesnât strike us as a different object, just a deformation.
Then again, trying to touch that object requires more mental work since the gradual deformation of our finger/hand as we place it in the water or behind the bottle is unusual. We would probably need to readjust our aim.
Also, Salvador Daliâs molten clocks are still clocks.
If we were shown the actual clock Dali used to model the painting after, together with different types of clocks he did not paint, we would have little trouble recognising the correct clock. Itâs that same clock, but deformed.
Another point is when we manipulate a long object, like a ladder or a broad ruler. If we drop it away from us in the long direction, the further part will scale differently than the close part. But if we drop it across our view, the scaling is different again. So scale must work differently for each dimension, or perhaps in a multitude of directions.
I donât think this invalidates Jeffâs point, but it would need a mechanism that includes directions.
I think each of us have a pretty good mental model of rigid body transformations (translation, rotation). So this becomes one of those allowed transformations where we can easily manipulate the appearance of an object in our imagination and recognize it under various transformations. The representations of that object in those orientations are sufficiently correlated within the network that they evoke very similar mental associations and functional responses.
As for the nonlinear transformations/distortions, it may still be possible to recognize individual features and the relative displacements between them. It may take a little extra time to make the connection at first, but the more often one experiences such distortions (e.g. looking through a water glass) the more it becomes one of the âallowedâ transformations that our brains quickly learn to compensate for and enable the appropriate associations.
Thatâs a good insight. But this evolution still needs an underlying dynamic system. Becoming allowed transformations requires some innate plasticity to cope with this kind of evolution.
Regarding your first post. (I had to read it a couple of times⌠sorry)
I think the SDR does not contain all information of one object. I think the SDR contains one (or possibly more) feature(s) and its relationships (relative distance?) to many other features.
The question is (I think) if the orientation of the related features (and therefore the orientation of the object) is encoded at this level in the SDR, or in another part of the brain.
I agree that the object itself is probably not represented by a single SDR, but rather a series of SDRs that combine (a bag of) features in different relative orientations (from the observers point of view). And I also agree that part of the learning algorithm implemented by the cortical columns is the ability to correlate all of these representations in such a way that they can be easily traversed by grid-cell like transitions (as the current theory suggests).
We can still recognize melting clocks as clocks, but itâs not the same as overall scale. You donât really notice different overall scales, like at different distances, whereas deformations pop out.
Maybe the mechanism for deformations is different. Maybe start with an overall scale, and then fragment a map of scales into smaller pieces until it determines the scaling on each part of the object.
It might be easy to scale along a line through space. As I understand it, each minicolumn does path integration along a line through space, by adjusting position at a rate based on movement speed along that line.
So to scale along that line, multiply path integration by the scale for minicolumns with the exact same line. The closer the minicolumnâs line is to being perpendicular to the scaleâs line, the less the effect.
Thalamus receives a motor copy signal, so maybe matrix cells use that as part of handling uneven scaling. Matrix cells also project down the cortical hierarchy (I think, not sure tho), which could be part of identifying different scales for each feature.
I think the mechanism that allows us to recognize rigid body transformations is essentially the same as the non-linear deformations. However, the nonlinear deformations look strange to us because we do not experience them as often as the everyday changes in position and orientation. If I see a melted clock, my first thought (after eventually recognizing that it is in fact a clock) is probably going to be, âWhat the **** happened to that clock?â Followed shortly thereafter by much mental conjecture on the possible circumstances that could have resulted in the specific situation I am observing.
Youâre might be right. One problem is that adjusting speed of theta oscillations might not work for uneven scaling.
Iâm trying to think of deformations we experience regularly. What about two cups, one stretched tall and one short, but equal width? I think those would feel like two different cups, whereas if I had one cup twice as large as another, itâd feel like a duplicate. Although, that reasoning feels like a stretch.
What is a âseries of SDRsâ if not itself just another SDR? What is the algorithm to create SDRs to âcombine a bag of featuresâ?
What is that âcurrent theoryâ wrt grid-cell traversal?
Introspection is not science. We should be able to demonstrate empirically that an animal can recognise objects through a certain transformation, then set out to replicate that as an algorithm on SDRs. Baby steps.
Thatâs because reverse engineering something isnât science. If you try to do that as if itâs science, you ignore the external motive you assign to something, but itâs still there. Youâre not trying to explain why you observed something about the brain (which is because it developed to be that way) but rather what purpose it serves. The external motive (hypothesized purpose) ends up being something vague like efficiency. Thatâs unscientific because itâs nearly impossible to disprove vague hypotheses about what something does.
Of course itâs essential to test hypotheses once you have them.
I agree, but sometimes introspection is useful for narrowing down the choices of ideas to investigate further. But also keep in mind that it is incredibly difficult to make empirical measurements of a functioning brain under sufficiently controlled conditions. To make progress you work with the tools you have available.
I see introspection to be of limited utility. Much of what the brain does has no cortical representation and as such, is not available to introspection.
Just for fun, try to bring saccades into your consciousness. The scanning and integration of objects is an observed behavior but try as I like, I canât âexperienceâ it.
Likewise, the loop of consciousness has a significant portion of the process in subcortical structures. We can draw a box about the âsubconsciousâ but it is not available to introspection.
Emotions are a subcortical process - we usually become aware that we are being influenced by these these structures by the effect the chemical messengers have on the body.
No, itâs because it just isnât. You canât use the science tools of observation, measurement, hypotheses, theory, experiment etc on introspection.
Reverse engineering is not science, itâs engineering: the rigorous use of known science and technology to produce a known end result, instead of new knowledge. I know, because Iâve done it.
HTM does not reverse engineer neurones and synapses. It assumes science and new knowledge, and then attempts to hypothese an algorithm, which is then implemented to be in software. What matters here above all is the data representations and the algorithms that operate on them. Whatever the computing model is, itâs quite unlike anything weâve done before, and thatâs the exciting bit.
I donât mean introspection can prove anything, just that itâs useful for generating and constraining testable hypotheses about the brain.
I think the idea is, if you arenât aware of something, it isnât represented in much of your neocortex. Lack of experience.
Weâre still not on the same page. Within limits, introspection has occasionally been somewhat useful in triggering ideas for experiments that might in due course lead to real science. It still isnât science.
Best to keep well clear, really.
âmore recent debate has questioned whether there is anything like a fixed toolkit of methods which is common across science and only science.â Stanford Encyclopedia of Philosophy - Scientific Method