Please discuss this paper below.

# How Do Neurons Operate On Sparse Distributed Representations? A Mathematical Theory Of Sparsity, Neurons And Active Dendrites

Has this been published in the meantime? It’s be almost 2 years since the first preprint.

Not yet

I have been working on handling reviewer feedback, which included some new derivations and running new simulations. It has been low priority (the Columns paper and associated work is higher priority) but I am slowly making progress. The main conclusions haven’t changed though.

I am troubled by the absence of considerations of “ambiguous input”,

i.e. overlap of multiple patterns, in this and the related spatial

pooler and sequence memory papers. Only “single pattern” inputs seem to

be considered and whether the system under investigation, single neuron,

spatial pooler or sequence memory produce the correct output for that

pattern. As per these papers, the single pattern presented on input may

have noise, defects etc. but a single pattern with noise or defects is

still a single pattern.

What about an ambiguous situation when let’s say the vision systems sees

something that could be the letter “B” or number “8” according to the

features, and only mechanisms higher up the processing chain, e.g. the

sequence memory or even higher cortical regions have the knowledge to

disambiguate between the two. Presumably the “B-or-8” pattern would be

represented as a superposition of binary vectors representing “B” and

"8", the way sequence memory has superpositions of possible next steps

(A-B-C vs. A-B-D example). Do e.g. single neurons and the spatial pooler

as a whole preserve this superposition so that it can be disambiguated

by the sequence memory.

Thanks

– Rik

Hi Rik,

Yes, this superposition of patterns is quite an important aspect of SDRs and HTM. The example you give of multiple predictions is exactly right - the TM can predict a superposition of next steps. It also shows up in pooling, where a single dendritic segment may represent many different patterns. In this case the synapses represent a superposition of all these patterns.

Many of the basic properties of SDRs are preserved but it’s easier to have mix and match errors, particularly with low dimensions.

We refer to superposition in our papers as the “union” property. In [1] it’s in Section 3.3, from the viewpoint of a segment that has learned a bunch of patterns. The same equations apply if you have a superposition of multiple input patterns. In [2] it’s discussed in Section 2G & H. It’s similar to the way Bloom filters work.

The TM can preserve this property (even for pretty large unions) and carry multiple predictions forward in time, but the SP doesn’t always preserve it if you have more than a few input patterns superimposed.

Thanks,

–Subutai

Hi @subutai,

Thanks, the “[2]” means which paper? I can’t find a section 2G & H in

either the spatial pooler or temporal memory paper.

In any case, “SP doesn’t always preserve it if you have more than a few

input patterns superimposed”, I like to see this experimentally verified

and quantified. I’ll look into doing this at some point. Surely this

ability to preserve superpositions depends on the model parameters

chosen. Here again we need a quantitative model – which will

materialize one day – where we can just compute the former given the

latter.

This is in the context of broadening the HTM model beyond brains into

machines and other territory. E.g. machine intelligence applications

might have a need to detect relevant patterns in thick soup of plausible

but ultimately irrelevant patterns. Think of following a conversation at

a cocktail party where all sorts of other conversations are going on.

Regards

– Rik

Sorry - forgot the references. I was referring to this one on arXiv:

Ahmad, S. & Hawkins, J. (2015) Properties of Sparse Distributed Representations and their Application to Hierarchical Temporal Memory.

http://arxiv.org/abs/1503.07469

I agree - it would be interesting to see it quantified. I think the main reason is that the SP is a competitive process that selects the best matching columns. If you are only selecting the top 40 out of 2048 (for example) you don’t have room for including too many superpositions. (You’d like to allocate at least 15-20 bits per pattern.) Yes, you may be able to improve this by changing some of these numbers.

Could somebody please elucidate equation (5) for me?

The paper says (page 5):

Eq. (4) computes the probability of a false positive, but what about false negatives? If a pattern

corresponding to one stored on a segment is corrupted with noise, it is possible that fewer than \theta synapses will overlap. Assume a dendritic segment D represents a subsample of some presynaptic activity pattern A_{t} using s synapses. Let A^{*}_{t} represent a corrupted version of A_{t} such that v of the ON bits are now off. If v is sufficiently small, i.e. v \leq s - \theta, the probability of a false negative is 0. As v increases the probability of a false negative, i.e. A^{*}_{t} \cdot D < \theta, increases. We can compute the probability of such false negatives in a similar manner as above, by using overlap sets. The number of vectors A^{*}_{t} that have exactly b bits of overlap with D is:\lvert \Omega_{D}(a_{t},v,b)\rvert = {s \choose b} \times {{a_{t} -s} \choose {v-b}} (5)

I can’t see why that equation should hold. For instance, if a_{t} = 1000, s = 20, v = 1, and b = 20, is the number of vectors A^{*}_{t} with exactly 20 bits of overlap with D equal to {20 \choose 20} \times {{980} \choose {-19}}?

edit: change the equation number at the top of post