Indeed, a single cell does not connect to all of them. It connects sparsely, only to those that are relevant.
To one from a subset of those which were active at previous step, when it is time for this cell to “predict” that it will fire next. This works because activations are sparse, and a subset of a sparse vector is sufficient to identify another sparse vector with huge probability. So… in effect, only those synapses are required to be remembered for the TM to learn a sequence of events.
Connecting all of them would be self defeating. For TM and… for brains.