Twitch was certainly made for you Matt, and your mission. I was finally able to see how you use WebStorm and various applications in action.
I was at my day job while you were streaming, but spent most of my Friday night (into Saturday morning) watching you code. For one of the bugs you had I expected resizing of the window area to have been the problem, as it can be in VB6. I was thankful to see that you soon noticed that possibility.
You led me to thinking about how feature memory plays back related touch, taste, smell and motion data in a way that we are essentially experiencing the stimuli all over again. Could that be the primary feature data?
In the moving shock-zone environment that I use important features are something felt, like bashing into a solid wall, moving freely at full speed on a comfortable surface or one that hurts their feet and should avoid, and confidence boost from making it to an attracting location where food reward helps eliminate hunger.
The behavior of (by becoming inactive) solid boundaries relative to the rest of the environment are mapped out in a way that would make a water balloon a roundish solid membrane where water motion swirls around inside. Water may squirt or burst out when some of the places containing the area are no longer containing wave action of a freely (as in sporting event stadium waves) propagating internal area behaving as water or gasses would. Drawing area is a hexagonal grid where features are placed according to their physical properties, which depend on how each column passes, reflects, blocks, or generates wave signals to its neighbors.
I focus on generating a theater of the mind, where signals act out the properties of what is being modeled. The river area is then contained by solid-like boundaries and has directional flow with waves on top that vary in height depending on other conditions. Water may contain something that attracts us, and we look into, while also avoiding edge or we at least wet our feet. When what attracts us changes, traveling waves point out new routes around obstacles, and favors the shortest path. There is still no knowing whether this is true for biology, but the method certainly works very well for getting around in an otherwise extremely hard to pleasantly navigate environment.
A nice thing for HTM theory is the need to predict ahead by at least 1/10 of a second. Interesting new article:
Without some method of predicting outcome of its own motion a fast moving ID Lab critter will race past its food, instead of ahead of time slowing down to a stop. For my purposes I coded a distance dependent circuit that decreases confidence level of motor actions that lead to being over the required speed limit for landing. There is no direct control of motor, just a memory bit that only becomes active just before something bad like that happens. This is enough for motor memory to self-organize actions accordingly. Since HTM makes predictions there should be an easy way to use that instead, but unfortunately I’m not sure what is most biologically plausible.
If the agent in a HTM system similarly has a motor system with four or more forward speeds that roughly double as force stays applied then the first test would be to not fly off the map and maybe crash the whole program, or forever mindlessly bash into containment walls.
Live rat data for the environment I use suggests that in this task each place is roughly the size of the animal’s personal space needed to freely maneuver, or approximately body length. A +1/-1 integer movement through environment at each 10+ HZ time-step would be traveling at a high rate of speed in an environment where ~0.01 displacement precision is required to position its mouth over a virtual food pellet.
If a coffee cup were mapped in with top edges and handle sparsely plotted using attractors then one or more articulated fingertips (in a map view of what is within reach) could in one fast motion get them all there in an instant, and afterwards have fine speed control over the surface.
An external environment could be drawn in using PyGame or Canvas, but of course what is being modeled in the brain has a hexagonally arranged geometry where objects have features most easily defined by how cortical columns represent objects in space. There is then a generic border/boundary for something impassable, and depending on temperature the edges of a cup are an attract or avoid fingers could try to navigate through to grab (new attractor) something cool that just fell inside.
That’s at least my best guess for how the spatially relevant information gets mapped out then acted upon. The rest of the data would be replay of past sensory experience including touching surfaces of a given temperature, in turn causing a cup to be mapped in as an attract or avoid.