That is a very hard problem, because the SP mashes all the features together. Each minicolumn represents multiple features, and from an isolated activation, you can’t tell which features are being represented. It is true you could hard-code a history somewhere so you can retrieve it, but that is going to be super compute-heavy and non-biological. This gets at the same issues underlying the Exploring the "Repeating Inputs" problem.
When minicolumn activates, you have to trace the predictive cells from the previous step to see the context in which in activated. This context could be one or many contexts combined (ex: sometimes E follows B, sometimes it follows C). These contexts are also within minicolumns, and activated because they were successfully predictive in the previous time step, now you have a tree of contexts that expands exponentially as you go backwards in time. It is the same problem whether the context is time or something else.