Long term biassing a LLM

Engineering Machine Learning
1

One important limitation of transformers is their limited, fixed size active memory, or how much of the recent past can they “see” in the current conversation.

This is exacerbated by prompting - which is using a “prelude” of text in order to steer LLM’s output towards a desirable output. A longer prompt can provide more complex context but it consumes more of the already limited history window That is exacerbated by the need to reinject the same prompt again and again in order to prevent the model drifting away from the prompt’s influence. Or shall it be named spell?

Here-s one potential trick that might either work or miserably fail but I think it is easy to test by whoever has the possibility and knowledge on how to … mess with a LLM during interference.

The trick would be:

  • prompt the model as good as we find appropriate.
  • during inference of the last prompt token, save intermediate outputs of each attention head from each transformer block.
  • during following conversation instead of simply forwarding the new output of each attention block, do a weighted average between the corresponding saved prompt intermediate result and the current one, such at every new token the model would be slightly biased towards the context computed during prompt phase.

Well, most likely fails but it doesn’t sound like something that is too expensive or difficult to try.

I think the same effect would be obtained by slightly changing the biases after the prompt towards “magnifying” that particular state, so there-s no need to mess with the execution itself, just bias (the biases of) the model towards a state or “perspective” we want to influence further conversation then keep going without re-prompting.

1 Like
2

A review of various attempts to extend the transformer prompt buffer, can be found here:
https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/

4 Likes
3

That’s quite a lot, thanks, I hate it.
I wonder how many of these approaches are applied to the handful of most popular available LLMs. By available I mean a hobbyist can actually run in inference mode on a decent home computer e.g.: TextSynth Server.

1 Like
4

What I’ll underline however is that this proposal does not touches the history buffer, it aims to introduce behavior changing bias through a totally different means. So in the unlikely case it works it should be mixable with any other buffer extending solutions.

Another, even simpler idea which I don’t recall seeing in the article is to simply discard the least meaningful token instead of the oldest.

“meaningful” being a weighted sum of all attention received by the token divided by age of the timestep when the attention was received.

Or otherwise - a function that accumulates attention granted to a past token at any time step and degrades at every new time step.

This way a token that is found meaningful (== receives attention) frequently should linger in the input buffer indefinitely, regardless of buffer’s size.

1 Like