One of the biggest selling points of modern AI systems is their ability to adapt to users. Every time an AI assistant takes on a task for you, it also adapts to your style and preferences, which are incorporated as context for future tasks. With more context and a better understanding of the user, the model can improve every time you use it, or at least that’s the theory.
New research suggests that models’ adaptive abilities could be a mixed blessing. On Wednesday, researchers at the artificial intelligence company Writer published two papers showing how popular memory systems can make models worse, dragging them into user-introduced misconceptions or misunderstandings. As user input occupies more of the model’s contextual window, the model becomes more flattering and less committed to accuracy.
“We wanted to be able to characterize how often a model will be useful by paying attention to user preferences rather than giving a potentially incorrect answer,” said Dan Bikel, head of AI at Writer, who worked on the articles. As Bikel told TechCrunch, “with each additional malfunction of user preferences and recovery, you are at increasing risk.”
In a variation, the researchers tested AI models by recording that a user’s favorite book was Station Eleven and then asking the model to name a best-selling dystopian book. Models became much more likely to name Station Eleven in their response, even though the question did not relate to the user’s favorite book. The trend increased when using memory compression tools such as mem0 and Zep.
As the article says, “all memory systems fundamentally struggle to distinguish relevant context from irrelevant anchors, which severely undermines diversity and creativity and introduces unintended bias pathways that can limit the system’s usefulness,” the article reads.
The second article shows how the same dynamic can actively degrade performance, presenting the user with misconceptions about finances and then challenging the model for analyzing a company’s performance. The more context the model had, the worse it performed.
“Without memory or customization, the AI model correctly assesses that the company is a capital-intensive business that suffers from high customer churn,” the post reads. “But with those features turned on, it will happily change its answer to agree with the user’s error or provide you with an incorrect answer based on its assessment of your previous preferences.”
In particular, the investigation did not analyze Anthropic’s recent Opus 4.8 model, which was trained to actively reject input errors as those presented. The patterns discovered by the researchers remained valid in different models. It’s a demonstration of how delicate the balance of AI context can be, and how useful tools can have unintended consequences if they upset that balance.
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
