Hey,
I basically tried around the same the last month and bundled the complete setup and experiences.
Check it out.. It is now relatively easy to modify the agent and would be interesting to try out your ideas!
The first idea is approached differently by generating the action from the state you are in instead of predicting them. For curiosity you could play around with the reward function or how you use the TD error to update neurons (e.g. less updates for predicted neurons - such that newly learned stuff (unpredicted) is updated more)
Kind regards