Interesting thoughts. Of course all we can do right now is speculate and dream about what true AGI might look like in the future – if it is ever realized. It’s comforting to think human consciousness can be explained through classical physics but I wouldn’t be completely shocked if that isn’t true.
In your hypothetical scenario, I assume an evolutionary algorithm (as they are implemented today) would be randomly mutating and editing parameters of a model based on a fitness function. That kind of implies the need for a “birth” and “death” of each individual of the population, sure. But how would we decide the individual’s lifespan? If it has a mechanical body, do we decide it should “die” when the materials that compose it’s body sufficiently deteriorate? But why couldn’t the parts just be replaced or it’s computerized brain just transplant into a new body? Moreover, once a successful design is reached, why not just massively copy and distribute it? No need for evolution anymore. That future just doesn’t make a lot of sense to me. And if the individuals of the population are completely virtual, handling that seems even less intuitive. Sci-fi has surely had a lot of fun with this the past few decades.
We know brains exhibit plasticity. Biological processes are slow and limited in flexibility at least compared to what can be done on a computer. Evolution is even slower. Who is to say AGI might just be able to start at an initial state and learn continuously to adapt to new environments. And since the individual would be non-organic, there’s no reason to include any concept of a natural death. It could just simply “live” forever outside of external intervention. Turing believed AGI would need to start as a blank slate like a child and learn through self-exploration and instruction. And again, because computerized brains would boil down to binary, that learning process would only have to be done once before it could be replicated and transferred into new individuals who might branch off to learn their own specialized purposes.