I’m not sure what you mean, but experience replay works as follows. First a bit of background.
In both the brain, and in deep networks, learning is not immediate. HTM abstracts the forming and strengthening of synapses into a single step process, but it’s actually a multiple-step chain of biochemical interactions that requires many repetitions of the pattern you want to learn. So each time a neuron sees a pattern, its connections are only updated by a small amount, and many repetitions are required to fully solidify the pattern detector. Deep networks are trained in an analogous way, where each presentation of a pattern only updates the synapses a small amount.
So to actually learn anything, you need to replay your experiences many times. In the brain, this is believed to be done in the hippocampus, which is capable of very fast one-shot learning, but does not store patterns for long timescales. As you sleep, and at other relevant times, the hippocampus replays your experiences and gradually consolidates those patterns more permanently in parts of the brain such as the neocortex.
In deep networks, this is done by keeping a buffer of past experiences during training time, and randomly sampling from this buffer for each training update. This is done for two reasons. Firstly, learning is achieved through small changes so you need to see each experience multiple times to sufficiently adjust your connections. Second, in order to avoid biasing your network toward only recent experiences and forgetting the past (“catastrophic forgetting”), you need to sample evenly across your past experiences.
That’s experience replay. It’s not “part” of the network and it’s not used at evaluation time, it’s just a training-time mechanism for evenly sampling the possible things you can learn.
Do you mean unsupervised learning? In supervised learning, you are mapping inputs to known outputs, like classifying digits. In unsupervised learning, you are discovering structure in the input data, and encoding it with useful representations, like an SDR encoder or the spatial pooler. And online learning just means you’re processing an ongoing input stream of indefinite length, and updating your model as you go.
Machine learning is capable of generalizing remarkably well, but it always depends on the training data you give it. There’s no magic algorithm that can learn things it wasn’t shown. The same is true of HTM, and the brain.
I strongly disagree with this. Deep learning as we see it today was mostly invented around 1980. The problem is not with engineering approaches to AI, but we just didn’t have fast computers and tons of training data back then. And before anyone complains about the huge amount of training data required, take a second to think about how much training data humans have. There are 30 million seconds in a year, and if we assume we process input at 10Hz or more, that’s hundreds of millions of training examples per year, and humans don’t become useful until after more than a decade of training.
If you think development in AI is very slow right now, then I’m not sure what to tell you. Look at a newspaper? And almost none of the researchers driving this development are neuroscientists right now, many don’t know or care much at all about the brain. I think the brain is a useful source of inspiration. But it is absolutely not certain that it’s the only way, or even necessarily the best way, to make working AI systems.
That kind of thing in academia can seem like bullying. We call it rational criticism, and it’s the reason science works. It provides incentive to prove it beyond a doubt, and that’s a healthy goal for all developing technologies. No one can argue with you when your technology is beating all the competition. Until then, more work is needed!
I highly recommend doing that. I’ve obtained dozens of useful insights in the process of understanding how deep learning works. Check out the universal encoder I posted a while ago. It’s written in TensorFlow which is powerful enough to build any deep network you want, and my code should be easy to understand.
It’s very hard for me to understand how the very cutting edge of AI technology, beating every benchmark it’s been tried on, solving problems computer scientists wouldn’t have dreamed were possible to solve even ten years ago, can represent the past. I can only assume that sentiment is coming from inexperience. Definitely have a look at the technology. Play around with it. And take what you learn and use it to improve HTM!