Yes, the problem is likely that there are too many distal segments / synapses. There is no trimming going on as new patterns are learned.
There is a way to counter this, but it is deep inside the Nupic code. What you would do is adjust logic within the TM so that anytime new distal synapses are created, the same number of synapses are randomly deleted from the global pool of distal connections (above some maximum amount of synapses).
This sounds easy, but once you go looking in
Cells4.cpp for how to actually do it, you see that is will take some serious testing to ensure you don’t break functionality.