After learning about HTM a bit and with a general background in ML, it seems to me that a central difference is that ML uses mathematical optimization to optimize an objective function, and HTM’s goal seems to just being fairly good at predicting sequences.

Intuitively, it seems to me like there’s an inherent trade off between generalization and optimization (I’ll be happy to hear about any attempts to formalize this concept) and this is possibly the reason why the ML algorithms of today really excel at a particular task (weak AI). The attitude of ML today is to set up a model with a big parameter space and to use gradient descent to optimize it to perfection. I fear that such approaches may be losing on the generality side.

HTM on the other hand seem to be less accurate than current ML algorithms but is possibly far more general.

Do you think this is a fair reading of the state of affairs?

Another thought is about how this applies to AI safety and alignment. Where the question is how to design an objective function such that it aligns with our desires even when the AI becomes super intelligent and can explore a much bigger parameter space in it’s optimization.

First of all, due to instrumental convergence, I think it’s fair to say that as the AI becomes better at optimizing a certain objective function it also becomes more general. Given the tradeoff discussed above, the AGI will then not be able optimize the function up to arbitrarily small error, thus possibly solving the AI alignment problem (we don’t have to worry about it going out of control because it finds a more optimal weird state of the world which we didn’t think about).

What do you think about this?