A little writing about neuroscience might soothe my mind today....
We use the term "temporal pooling" to refer to a process whereby a set of neurons stays active even though the input patterns to the set of neurons is changing. Temporal pooling is a many-to-one mapping; many input patterns map to a single output pattern.
This type of process is well documented in neuroscience. For example, as your eyes move the input pattern to the neocortex is constantly changing yet several levels up the hierarchy we find neurons that are stable and specific to the object being viewed. Think of it this way, the ability to classify a set of input patterns as "something" requires temporal pooling, so we know it is occurring wherever classification is occurring.
Using feedback projecting in the opposite direction, a single "higher level" pattern will invoke a set of input patterns. Typically the backward projection does not activate the input neurons but only depolarizes them, to bias them.
In HTM today we don't model individual spikes, neurons are actively spiking or not. Temporal pooling only requires individual output neurons to recognize multiple input patterns. It is just a many to one mapping. As long as one or more of the input patterns is active then the output neurons will stay active. I don't see any timing issues with this. Let me know if you are still concerned about timing issues.
We believe temporal pooling is being performed on both high-order sequences such as temporal memory, and on sensory-motor sequences which do not occur in a predictable sequence, but instead depend on what movements you make. As you might know we are currently working on a paper and simulations on how the basic mechanism used in temporal memory can be applied to sensory-motor inference. As part of this work we have started to model, in detail, temporal pooling over sequences of of sensory-motor inputs. The results so far are encouraging in terms of capacity and performance. In short, it seems to work pretty well.
Where in the neocortex does temporal pooling occur? We believe it occurs in several places. At the moment we are modeling L2 pooling patterns in L4. L4 changes with each movement of a sensor, such as a finger or the retina, and L2 temporal pools to create a stable and unique representation of the object being sensed. I also believe L5a is temporal pooling input patterns from L6a. I used to believe temporal pooling occurred between regions in the hierarchy. It still might. But I am now very confident that temporal pooling is occurring several places within each region. Therefore temporal pooling does not HAVE to occur between regions. As long as it is occurring somewhere in the feed forward path up the neocortex we are good.