This is my first post so I’d first just like to say thank you very much to Jeff Hawkins and all the researchers in HTM theory for everything you are doing and especially being so open with your work - its really quite amazing.
I’m interested in vision, how it works and how we might best create basic artificial vision systems. Specifically, I’ve been lately looking at how the eye’s motion supports vision perception and creating artificial systems based on this - Optomechtronics.
I have watched many of the short videos on the sensorimotor hypothesis and have been thinking about how vision relates to it. While I’ve seen vision mentioned in your discussions, it seems you prefer to reference the ideas to touch and I can understand why. But it also gets me thinking about the many analogies of vision to touch (and the differences).
Skin has an uneven distribution of sensor density – fingertips have a lot, less so elsewhere and we have to explore with fingertips to get a high acuity measurement. The Retina’s foveal region are like your finger tips with much high densities of photoreceptors than the periphery. We explore visually through movement of the eye. The eye has the rods and cones so that’s a difference but one that doesn’t seem that significant.
What follows is my understanding of how it works
Saccades are mostly conscious rapid movements (e.g to words in a sentence or faces in a scene) to re-position your foveal region prior to a fixation.
During fixation, your eye performs the subconscious relatively slow drift movement which is so that your photoreceptors can generate output due to the change in light flux as the fixation point in the scene is scanned across the fovea. Microsaccades occur to reposition the fovea back to the fixation point so that another drift can be performed – perhaps with a different velocity vector to transduce a different aspect of the scene. Horizontal drifts transduce mostly vertical edge information and vertical drifts transduce mostly horizontal edge information etc. Question – If this is true, is there a feedback mechanism that influences the drift direction (do the neurons ask the eye for a particular type of information)?
I’ve read that the V1 neurons are inhibited during the fast eye motions (saccades and microsaccades) so that to me implies a feedback. This suppression during those movements also helps separate eye induced motion from scene induced motion. This and other functions like recognition (supporting ocular motion feedback and anticipatory excitation) must all happen at a very low level in the layer hierarchy which is why I was excited to hear the same type of hypothesis in the videos. There is also the ocular micro tremor (OMT) that occurs during motion somewhere around 80 Hz I think – and I’ve read that this is likely just an artifact of the mechanical control of the eye motion and not important in vision itself.
So, I am very interested in learning about HTM, the available implemented models and how I can apply it with my setup. I can simulate most eye motions precisely and generate simulated spikes (responses) from photoreceptors – I perhaps can discuss this setup at a later date but it’s not that important to the topic.
Each response from each photoreceptor has associated with it precise timing information. This timing is fairly important, especially when considered in relation to the movement velocity (speed and direction) and even more so when aggregated with precisely timed information from neighboring photoreceptors. Higher contrast transitions generate responses earlier and more often than lower contrast transitions. And so the relative timing of the responses encodes important information about the scene. I think with just one short drift motion, a very accurate representation of the scene is captured in a very sparse representation due in large part to the precise timing and sensorimotor boosting; aided by the anticipatory response that occurs due to scene familiarity etc.
I’ll be very interested if/when timing will be added to the HTM but until then I have a lot to learn anyway. If anyone has come across or knows of good resources for these topics, I’d really appreciate hearing.
If anyone has made it down to this far – then thank you for reading and I would love to hear your thoughts. Even if they are thoughts like – that’s completely crazy, or that’s completely obvious! Because I’m not well educated on these matters but have just become very interested. My education is in Systems/Cybernetics started at a machine learning company back in the 80’s and have been a Systems and Software engineer mostly in electro-optics – hence my interest in vision.
Hoping to retire in the next few years and make this my hobby….