One of the biggest selling points for modern AI systems is their ability to adapt to users. Every time an AI assistant takes on a task for you, it’s also adapting to your style and preferences, which are incorporated as context for future tasks. With more context and a better understanding of the user, the model can get better every time you use it — or at least that’s the theory.
New research suggests that models’ adaptive abilities might be a mixed blessing. On Wednesday, researchers at the AI company Writer published two papers showing how popular memory systems can make models worse, pulling them toward misconceptions or misunderstandings introduced by the user. As user input fills up more of the model’s context window, the model grows more sycophantic — and less committed to accuracy.
