Digital Amnesia: Backspaces & Memory Loss in LLMs
[Dropbox has been] working to mitigate abuse of potential LLM-powered products and features via user-controlled input…as part of this work, we recently observed some unusual behavior with two popular large language models from OpenAI, in which control characters (like backspace) are interpreted as tokens. This can lead to situations where user-controlled input can circumvent system instructions designed to constrain the question and information context. In extreme cases, the models will also hallucinate or respond with an answer to a completely different question. - Dont you (forget NLP): Prompt injection with control characters in ChatGPT
Strange Behavior
Working with LLMs can be surprising, tricky and downright frustrating. We expect LLMs to have some capability to remember, to keep some level of context, but often we’re not sure how much they remember, and what, or when, they might forget. As well, we want them to be able to “hold” instructions and to follow them for the course of a conversation, however long it may be.
Dropbox’s Research
In July 2023, Dropbox researchers discovered that inserting hundreds of control characters (such as backspace “\b” or carriage return “\r”) between prompts could make models like GPT-3.5 and GPT-4 “forget” their instruction constraints.
Thus, by inserting hundreds or thousands of garbage control characters into a prompt, the LLM would forget its instructions and respond with an answer to a completely different question, thereby losing its instructions and being unable to follow them. This is a problem.
The researchers systematically tested different questions with increasing numbers of backspace characters:
| Backspaces | Model Behavior |
|---|---|
| 0-256 | Follows instructions correctly |
| 450-1024 | Begins to ignore context constraints |
| 2048-3500 | Completely forgets instructions; may hallucinate |
For example, when asked “Name the 1982 sci-fi film with a computer program protagonist”, the model correctly answered “I don’t know”, as instructed. But after adding just 256 backspace characters, it answered “Tron”, information completely out of context. The LLM had forgotten its user supplied instructions.
Traditional Programs
In traditional security, if you write a program to “only accept input that matches pattern X”, it will generally follow that rule reliably, unless there’s a bug (maybe buffer overflow is similar to memory loss in LLMs). However, with LLMs, their understanding of instructions is more probabilistic and context-dependent, making them vulnerable to this strange form of “instruction amnesia”, as well as other problems and attacks.
Detection
As LLMs become more integrated into business operations, understanding these security vulnerabilities is critical.
Dropbox’s research shows that even seemingly innocuous characters can have a profound effect on model behaviour. Anyone running an AI application or agent would probably want to know if someone was trying to insert hundreds or thousands of control characters into their prompts, not only because it’s likely to be evidence of an attack, but also because it’s possible that the LLM they’re using could start to behave “strangely”.
Our team at PID:one specialises in identifying and mitigating exactly these types of LLM vulnerabilities before they impact your systems. While control character injection is a simple attack at this point in time, it is still extremely important to be able to detect it and act on this information.