I agree we can ignore some connections, or at least save them for understanding later. Layers process things somewhat in parallel and connections can do things like add context.
I see the problem as figuring out what is just a specialization, what is universal, and what is universal but implemented differently. Physical structures like sublayers for different submodalities are clearly probably not universal, but thalamic matrix cells target L5a in rats and L2/3 in primates, and I have a hard time believing both of those projections are specializations. Having two implementations of an algorithm is useful for hypothesis testing. Specializations can give clues to the core algorithms since they need to plausibly make sense in context of those algorithms. It’s also important to find exceptions to supposed rules and see if there is an explanation.
Some connections are simple but don’t make sense. Getting the ideas right is very very very hard.
For example, layer 5 is usually thought of as just a motor output, but there are many reasons it isn’t. It even targets the first subcortical stop for some sensory inputs (a trigeminal sensory nucleus for whiskers, not the primary thalamus, but you could argue that too in one case). You could still argue it’s just a motor output.
L6 is usually thought of as sending feedback signals to the thalamus, but the deepest part (or in some species, cells in the white matter below) also seems to send feedforward signals to the thalamus up the hierarchy. You could argue that’s just a remnant of development, though. You could also argue those are matrix cells or it has something to do with them receiving direct sensory inputs.
One author speculated that the thalamic nucleus POm drives activity in the septa in barrel cortex, and eventually that became taken as fact maybe for a decade until it was found to be wrong.
Hierarchy is usually thought of as entirely cortical or through the thalamus, but secondary cortex and probably others receive direct sensory inputs in rodents and a little in primates. But the response during those sensory inputs might usually get inhibited by the zona incerta.
It’s an endless rabbit hole of caveats.