I have read the foundation papers listed in the Numenta.com site. Some HTM papers draw heavily from the SDR theory. Some papers focus on the theoretical properties of SDRs and sparse representation.
Throughout this forum, I see repeated references to the properties of SDRs and the assumption that HTM automatically gains from all the possible advantages of SDRs.
Here is where think that this is wrong: the biology does not work that way.
- In most HTM models the cell bodies can make connections with any input bit without restrictions. The does allow an HTM model to gain some of the advantages of SDRs but NOT a global union property unless some outside agent examines the connections.
- In pyramidal cells, the SDR is actually formed by the dendrites emanating from the various regions (proximal or apical) They pass by a fixed set of axonal projections and columns and certainly extend no more than 8 mini-columns distant from the cell body. This makes the SDR interactions local to the cell body.
- I cannot see how a global SDR union can be formed by this biological arrangement. I have never seen any mechanism like this in the brain. The closest it gets is if a dendrite has two or more learned SDRs. Partial activation of each of the learned SDR synapses could sum to firing potential. This would be local to the single dendrite with the input feature space limited to the small set of minicolumns/axon bundles that dendrite can reach.
- If a given cell body is to form a relationship between features the encoder will have to distribute the feature bits to a wide variety of receptive fields to allow the mini-columns to sample these features. This is a complication that is usually ignored in encoder creation. This also means that when examining what is going on in the biology this important property is overlooked.
I see that breaking away from the biology makes it easier to work with. For example - allowing the reach of a cell to be the entire input field makes the encoder task easier as you don’t have to worry about the spatial distribution density of the encoded values. This is what I often refer to as “well shuffled” data.
I have to balance that with the guiding principle that making models of the biology informs research into the biology and findings in the biology informs modelmaking. I read many papers on cell biology and over and over I find myself thinking - HTM does not work that way. The restricted geographic scope of both the mini-column and encoder means that there are important biological features that cascade to computational impacts.
An example - the original ANNs were simple summation/threshold units. Minsky showed in the book “perceptrons” that this had important theoretical limits. The first AI winter was the result. When researchers recognized that cells saturated and were arranged in layers these theoretical limits fell away. The PDP books were some of the first to show that by adding the sigma response curve and layers the models were vastly more powerful.
Lest you think I am bagging on HTM - NO - I think that HTM and the Deep Leabra models offer the strongest way forward in the search for a platform to deliver strong AI.
I am a firm believer in the Numenta published philosophy of faithfully reverse engineering the brain. I am encouraging other HTM researchers to spend some time learning how the biology works and try to gain from the lessons nature has to offer. When it comes to making a functional intelligence the brain is the only system around that has the "been there, done that” tee-shirt.
