Although AI companies keep building bigger context windows, bigger doesn’t always mean better. Researchers have found a strange pattern in long-context AI systems. Facts placed in the middle of a long document often get missed. Facts near the beginning or end fare much better. This happens even when the whole document fits easily inside the model’s window. Experts say it’s not a size problem. It’s a structural one. Attention inside these systems isn’t spread evenly across the text. This “lost in the middle” issue shows up in more than just test scenarios. It affects retrieval tasks, question-answering, and even code reasoning.
It’s not a size problem. It’s a structural one—attention simply isn’t spread evenly across the text.
There’s another problem, too. As inputs grow longer, accuracy can drop, even on simple tasks. Experts call this “context rot.” Redundant or similar-sounding passages can confuse the system. This can lower accuracy across the entire document, not just in the middle. A bigger window means more room for irrelevant information to sneak in. And that extra noise can hurt decision-making.
Benchmark tests back this up. Some long-context systems show solid results in retrieval tasks. But few keep accuracy above 64,000 tokens. Some models drop off sharply after 32,000 or 64,000 tokens. Even systems built for one million tokens don’t always perform well at that length. Scores at the one-million-token mark vary widely between models. This shows that window size alone doesn’t predict reliability. In one 20-document study, GPT-3.5-Turbo’s accuracy fell from 75.8% when the answer appeared first to 53.8% when it was placed in the middle, a drop attributable to attention bias favoring the beginning and end of inputs.
Adding more text can help models find information. But it also adds more chances for mistakes. Important facts can get buried under distracting details. Some systems start relying on memory-like guesses instead of checking the actual text. Others summarize instead of pulling exact answers, which can cause errors.
Needle-in-a-haystack tests, which search for a single fact, don’t always reflect real-world use. Basic searches can succeed with short inputs but fail once more distractions appear. Multi-fact tasks are especially hard. Performance can also drop sharply in lower-resource languages. Finding the right information doesn’t guarantee that a model will use it correctly. As context grows, so does the risk of failure. Even top-performing systems like GPT-4o have shown accuracy falling from 99.3% to 69.7% when moving from short to long contexts on benchmarks designed to require genuine inference rather than literal word matching. Addressing these structural failures will require responsible AI development and potentially new regulatory frameworks to ensure that deployed systems meet minimum reliability standards across varying input lengths.
References
- https://memx.app/blog/lost-in-the-middle-long-context-fails/
- https://www.wispaper.ai/en/research/long-context-language-models-reliability
- https://aclanthology.org/2024.mrl-1.18.pdf
- https://proceedings.neurips.cc/paper_files/paper/2024/file/c0d62e70dbc659cc9bd44cbcf1cb652f-Paper-Datasets_and_Benchmarks_Track.pdf
- https://arxiv.org/abs/2411.03538
- https://aclanthology.org/2025.findings-emnlp.301.pdf
- https://arxiv.org/html/2510.05381v1
- https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf
- https://arxiv.org/html/2501.01880v1
- https://www.databricks.com/blog/long-context-rag-performance-llms
