Ideal density of SDRs

#1

When showering. I thought of something being really strange. I recall that Matt said that ideally SDRs are ~4% dense somewhere in HTM School. But in practice I find that SDRs are mostly ~10% dense (Maybe caused by TMs predicting all possible future instead of one possible future). For example. In my Grid Cells experiment, the input SDR is 12.5% dense and the predicted SDR has a density of 17%. And in my NLP experiment, although the input SDR is ~3.5% dense. The generated SDR is 9~15% dense. And in my ECG experiment. 4% density is way too low to be usable, sending a SDR with ~10% density yields a better performance.

Have I remember anything wrong? Or maybe is 10% a better value?

1 Like

#2

I don’t have an answer but what seems to be the symptom when the sparsity is too low?

Is there a continuum where you can some some factor improve as you vary this value?

Going the other way - what happens when you increase the value past the point were you see “good” numbers? I am guessing over-fitting but would you know that with the work you have been doing.

0 Likes

#3

Thanks. I should have explained myself better.

In all of my experiments, when the sparsity is too low (ex: 4%). The anomaly score goes down very slowly or barely goes down. And sometimes the anomaly score oscillates between 1 and other values.

While when the sparsity being too high, the network never detects anomalies.

1 Like

#4

@Bitking, Any idea on what is going on?

0 Likes

#5

We settled on 2% sparsity because that is generally what is observed in the brain and because it worked for our experiment, so we found no reason to update that target sparsity. Your mileage may vary, please experiment and report any findings.

1 Like

#6

Ive used a wide range of sparsities:
Grid cells can have very large sparsity of 30%.
SDR Classifiers work best with low sparsity of around 1%.
The TM needs enough sparsity that its dendrite segments can reach the predictive threshold. But TM sparsity is also controlled by the SP sparsity and the num cells per minicolumn.

For information transition, 2% sparsity is typically pretty good, but it all depends on the application.

1 Like

split this topic #7

A post was split to a new topic: Question about Sparsity in Local Inhibition

0 Likes