Science starts with observation. The next step is formulating an hypothesis, some intuitive explanation based on experience in the observed domain. A later step is creating an experiment, predict results and generate enough data to show the results match the prediction. Only when enough results warrant the confidence, a theory is formulated and offered for peer review. If possible, this theory can be based on more fundamental theories, and cross-referenced to other scientific domains. But this is not always available.
Yet, all this is science. Including the observation. Ergo introspection.
You make my case for me. Introspection might give you a starting point, but itâs the experiment, theory and predictive value that interest me ie the science that follows.
You might think that you ârecognize this is a pipeâ but you donât. You recognize that this is an image of a pipe. You would never try to pick this up, or else youâd get fingerprints on your computer screen!
In a similar same way: if you had two coffee cups of different sizes, you might think âthese are the same type of objectâ but you will still think of them as distinct objects. Otherwise you would be unable to manipulate them correctly. Mr Hawkins thinks that he can pick up both cups with the same muscle movements, by only scaling the magnitude of the movements by a single scalar factor. He even proposes a specific mechanism to do this (altering the Thalamic theta frequency). I disagree, here are some specific examples where I think this hypothesis fails, try it!
Try grasping the cups: a smaller cup requires moving your fingers further to fully constrict around the cup, since it is smaller in diameter.
Try lifting the cups: the large cup requires more force since it weighs more.
Try picking up the cups by putting your finger through the loop of the handle: on a smaller cup you might only fit 1 finger through the loop instead of 3.
Try writing your signature small and large: see if you use the same muscle groups. Observe: when you increased the size of your signature did you also lift the pen up proportionally higher above the page? The hypothesis that scale is controlled by a single scalar factor would cause you to increase your vertical motion in proportion to the size of the image which your drawing.
In summary I think that:
Objects of different sizes require fundamentally different motor behaviors to interact with.
I think itâs a scaling of motor vectors, maybe in object-space. Other things have to translate that to muscle movements. Probably a lot is subcortical and in the where pathway.
If you have two exact duplicates, you have to keep them separate. Same object identity but different objects.
True that it has to represent object scale somehow, as in scale in the world not the change in motor vectors.
How can the neocortex represent objects @ orientations?
In the context of 3D geometry a location is three distances (R3) and an orientation is three angles (R3).
From the point of view of a cortical column in the visual cortex, what is the difference between location and orientation?
They can both totally change the visual appearance of an object in numerous arbitrary ways. Both can be controlled through muscle movements. It seems to me that orientation is a special type of location information. Do the theories describing locations not also work for orientations?
The book âperceptronsâ showed convincingly that layers of units could be replaced by a suitably constructed single layer. This had limits such as being unable to perform the xor function no matter how many layers you used.
This book effectively killed research in the area for a long time.
True believers kept at it and eventually worked out how to surpass this basic limit.
Adding the limiting activation to the mix allowed transcending the limits of the classic perceptron. It was now possible to form islands of meaning between the layers.
Much the same benefits with layers of HTM modules. Adding the H to HTM radically enhances the representation and computation possibilities. Itâs not the same thing then - it does more. The SDRs are now able to pool, both spatially and temporarily and these pools to be sampled to form conjunctions of these semantic meanings.
I never found perceptrons convincing and always saw that entire body of work as a blind alley.
I donât find âlayersâ a helpful concept. There are anatomical layers, but they are an artefact of the way that the cortex has evolved. And I havenât seen anything to convincingly justify the H in HTM either.
In my view the broad organisation of cortex into columns speaks to multi-processing. Large numbers of columns doing the same processing job requires a common data representation, and SDRs fill that need. The representation of sequence memory that learns by prediction as âstackedâ SDRs is credible. Something similar seems to apply for location, although I donât find it as convincing yet.
Rather than a hierarchy or layers I am left with a mental model of SDRs that:
in the sensory areas represent raw sensory inputs and successive refinements
in some higher centres represent more abstract properties, concepts and plans
in the motor areas represent broad motor intentions and successive refinement down to individual muscle movements.
Itâs SDRs all the way down. But this is a computational model and while we have a data representation (totally unlike any computer we know) we do not have an âinstruction setâ (also likely to be entirely novel).
Iâm guessing there will be 10-100 âinstructionsâ that represent ways of generating SDRs, of which we know of or can guess at a mere handful. Iâm guessing both SDRs and âinstructionsâ will be found in far simpler brains, from which the cortex has evolved. Thatâs where real science comes into its own.
Iâm sure that the deep learning people will be shocked to hear that!
Seriously- that branch of research captures some important properties of the brain.
HTM captures some different properties.
The brain does use elements of both technologies and they are both worthy of study.
As far as localization of function- there has been some very fine-grained work in this area. If this interests you I would suggest looking into the work on the connectome.
I will add that the preservation of topology and the hierarchy of maps in a unifying theme though much of the cortex. This allows the possibility of a spot in an early map to project to a higher level and still be meaningful in processing at that higher level.
The first speaker in this series (Marge Livingston) points out implications of this in her talk on Category based domains. Pay special attention to the bit about how information that goes DOWN the hierarchy trains the lower levels, even if the related sensory stream is not present. The implication is that the connection scheme is critical to forming categories, and not as much the content as is normally assumed.
By dead end I mean: not on the path to AGI. ANNs and their ilk are astonishingly successful at recognising and classifying, given large amounts of labelled training data. They can do things no biological system ever could, but itâs still a dead end. And no, I have no reason to believe they capture important properties of brains. Processing sensory input: just maybe, but beyond that: no way.
The connectome tells me just one thing: brains are packed full of neurones, which are deeply inter-connected. I read the book: it takes us nowhere.The basic precept is wrong: we are not our connectome.
That video is 4 hours! Sorry, but if there is something relevant in there youâre going to need to pinpoint it for me.
Iâm a software guy with a medical/scientific background. I see multi-processing, a data representation and a storage mechanism, and I look for the software, the instruction set, the programming language. People without my background think that neuroanatomy and connections will get us there, but theyâre dead wrong, thatâs just the hardware. If I give you the full wiring diagram of the computer you are using right now, you know nothing about what it does or how it works. Thatâs software.
So go find the software for a maggot brain and weâre on the path to AGI. Sooner or later weâll leave the ANNs in dust.
What I donât find quite convincing in the argument âreproduce the hardware and it wonât do anything, you need to understand&replicate the software tooâ is that software happens in the realm of physics too, it isnât projected from a platonic ideal realm.
If you actually make an accurate copy of a computer you will have to include charge states on its SSD or magnetization of its HDD. And there is its software! It will work without having to understand how it works.
If you replicate a brain you will replicate synapse position, size, and whatever is relevant to a synapse state, and there is its software.
What is funny is there are greater chances the brain replica would be a functioning one since it is much more robust against losses and noise. With lots of errors it will exhibit dementia but computer replica would not work at all with very few replication errors
Marge Livingston is the first speaker in the video, you can safely skip the rest.
In ânormalâ computer programs you have data and the sequence of operations that are performed on that data. You have a collection of locations where the data is stored and you specify the sequence of those operations. Data in a stored-program computer is accessed sequentially and the operations performed are sequential with the output from one step being the input to the next step.
In the brain, data and operations are mixed together. The connection ARE part of that computation. As you indicated, the pipelines of data are all run at the same time. The processing is still sequential, but massively parallel.
Since the brain connections are essentially fixed, data is not accessed and stored, it flows from one processing stage to the next.
The connections are part of the algorithm.
For this reason, I see that understanding the connection scheme is a critical part of understanding the computation.
For the basic computations I have compiled a list of the operations that I can see that the CC is doing here:
Lets say you have a coffee cup. And then you have three pictures each showing one cup. One shows a cup in another color. One shows a cup with a slightly different shape. And the third one shows exactly the cup you have in front of you.
You recognise that third cup. Even though it may look smaller on the picture; even though the ear is facing another direction; even though the picture is taken from a different angle and with other lighting conditions, you still recognise your cup.
I am also in the âcollection of featuresâ camp.
I see that in the association regions the stream of feature & displacement is the thing that accumulates to object recognition.
The visual scanning primitives have been reliably documented in many studies of saccades.
This subconscious saccade mechanism force-feeds a constant sequential stream of feature and displacement data to the visual stream to be assembled into object recognition. This makes use of the spatial and temporal pooling mechanisms that is thought to be the key operations of the cortical column computation.
As far as recognizing the image on the screen as not an actual object, I think that is a learned thing.
I think that until this screen thing is learned a pipe is a pipe.
Or an ant is an ant.
I donât have this in a maggot example, but in this case, I do have a frog:
Physics has no software (that we know of). It represents a set of more-or-less inviolable rules to get from one state to the next, and the next state is probabilistically determined.
Brains are different. They receive inputs and decide upon outputs. The decision process is modified by previous states ie memory. It takes software to do that.
Computer state is indeterminate at the quantum level. As Iâve said many times: the past is immutable, the present is chaotic, the future is probabilistic. Your thought experiment fails: it cannot be done, you cannot replicate quantum states. And even if it were, you understand nothing about how a computer works.
My point is: you do not see what I see. To me it is blindingly obvious that we know a lot about the hardware, a little about one particular data representation (SDR) and almost nothing about the software. You donât see it, but itâs there.
What you call ânormalâ is a reference to the von Neumann architecture, and there is no reason to believe brains use anything like that. So forget that whole line of reasoning, it doesnât apply.
At some abstract level:
NPU is a neuronal processing unit, possibly a column, consisting of hundreds of neurones. NPUs are general purpose: they are all the same (within a region).
Connectome includes all the connections inside a NPU, and the connections between NPUs
SDR is an atomic unit of data exchanged between NPUs. Its physical structure might be represented as (say) 15 ON bits and (say) 200 OFF bits.
An NPU instruction determines the output SDR: which output bits get switched ON in response to some input given the current NPU state. One example that comes to mind is a clock generator: an instruction that would cause a general purpose NPU to emit an SDR at regular time intervals.
As wild speculation, we might find the instructions in junk DNA. DNA is the instructions for generating proteins, so it has form in this role.
Thatâs what I see. If you donât, itâs because our perspectives are so different.
Are you doing anything with your special knowledge or is this just an exercise in letting everyone else know they are somehow doing things wrong?
If you have anything concrete to offer as to what people should be doing it would be welcome. Telling everyone that you have special insights and that what everyone else is doing is wrong is not constructive.
What is it you want me (and others) to do with your special knowledge?
Sorry, wish I could help. Iâm an engineer, which means I canât do the research but I can use it to do useful things. This is what I see, but now all I can hope is that others see it too, and go looking in the right places.
