I generally agree with this framing. We may likewise want AGIs to balance multiple needs—e.g. a need to respect cultural norms and a need to follow the law and a need to earn lots of money for the programmer. The human brain balances multiple needs, but not in a particularly reliable way. For example, we may go overboard satisfying Need A at the expense of Need B and then feel regretful about it. Or there’s the “getting addicted to drugs” example we’re talking about.
In reinforcement learning (and economics I guess) I think there are mathematical theories about how to optimally balance multiple objectives. The brain does it in a more ad hoc, unprincipled way. I imagine that if you set up 100 nominally-identical brain-like AGIs with the same set of a few dozen “needs”, they would wind up trading off among the various needs in 100 different ways, for random and unpredictable reasons. (I could be wrong.) I don’t know how you would solve that problem. My best guess is “we should make the ‘subcortical needs centers’ do more sophisticated processing / calculations than they do in the brain”. They are pretty dumb in the brain, or at least that’s how I think about it. There’s plenty of room to make them more sophisticated. But what exactly they should be calculating, I don’t know.
I don’t think habituation is helpful for this particular problem (if it is indeed a problem), unless I’m misunderstanding. In fact, it seems to work the opposite direction. The way I’m thinking of it is: let’s say I have a need for fame. I get a little bit of fame, and then I get habituated to that, so then my need for fame manifests as a desire for a lot of fame, and then a ton of fame, etc. So it seems to me that habituation works against balancing multiple needs, and instead makes behavior potentially more lopsided. But I dunno, maybe it’s different in different cases.