In NuPIC the TM does what you’re talking about, by using the Connections class. The connections class keeps a list of input axons and where all of their synapses go to, and it efficiently distributes each active input by iterating through just the synapses which the input connects to.
The SP does not use the connections class, instead using sparse matrices. In the community fork I’ve changed the SP to use the connections class and also optimized the connections class.
I agree. You need both. But start from the bottom 
From there you need “high-level” observations/intuitions to constrain your infinite search space. Using the top as the only drive, you end with DNN v2.0 .
Using bottom as the only drive, you end with the Human Brain Project (BTW: I suggest you take a look to edX course of Markran Simulation Neurscience. They are an incredible work… but I don’t know if the will succeed only following that route).
Thanks for the paper. I read some parts some time ago. Without investing too much time in that, my two cents is that is way too complex (and too specific). It should be easier and valid not only for vision but for any part of the cortex. I like more S.M. Sherman papers 
A post was merged into an existing topic: Deep Predictive Learning: A Comprehensive Model of Three Visual Streams