That picture is actually spot on Only the labels need extension.
The segments of the cells which control the predictive states. So these are segments that you drew below the cell on the top left side. There may be multiple distal segments. These segments are located at distal dendrites. That is why I kind of interchange segments with dendrites from time to time.
The segment which controls the active state of the column. There is only one proximal segment per column which is actually referred as proximal dendrite. Spatial pooling is actually the name of the functional process that involves the proximal dendrite.
Empirically you set higher permanence increase values than decrease values for the proximal dendrite synapses. Nupic does this and I am on the same page. As you have observed, if those two values are set as same, it becomes harder for a column to specialize onto specific patterns while the patterns are changing constantly.
The increase and decrease permanence values for the segments of the cells (distal segments) can be set equal. Nupic does this. Or the increase value could be higher just like proximal dendrites, I do this.
If we did have a single distal segment per cell you would be right. But what if we wanted a single cell to be receptive to different patterns. A cell can be predictive for different patterns involving a different subset of neurons. If all connections were in the same segment (or no segments at all) then how would you identify different patterns? Cells that belong to pattern A would stimulate the same segment that belong to pattern B. So for example insufficient amount of neurons (below the activation threshold) from both patterns would actually put the cell into predictive state. I hope it explains the need for multiple distal segments. As you know, proximal dendrite does not have segments.
Reading this made me think that you have not read the official algorithm in depth. If you haven't yet, try to go through this guide that explains all the algorithm steps of HTM which would fill in a lot of gaps.
If I did not misunderstand, what you describe is exactly how Numenta creates newer distal segments on cells and it is called temporal memory. That is why I recommended an in depth read.
This link contains a fine collection of videos here and there.
Hope this helps.