I have a half-baked idea for a reward function for curiosity. I will admit it’s kind of sketchy but I imagine that the agent will have some sort of mental space. You could have some need to fill or extend this space with items orthogonal to the current contents - a desire to learn something new.
When I say orthogonal what I mean is referring to the 2D nature of how memory is mapped in the cortex. In whatever the axis’s the particular map holds - areas that are not stimulated by a new memory (surprise) will want stimulations.
I have stated elsewhere that perception is active recall and that you learn what is surprising - the reward is proportional to both the amount of surprise and the amount of memory engaged in the recall.
The critter should want to examine it’s environment and learn new things about it, expanding it search space. If you balance that with programmed forgetting you should be able to establish some fixed territory.
I did say this was kind of sketchy?