Research Digest
arXiv BreakdownAgents & Evaluation · Research DigestThe Best Models Track State Well. At 10,000 Tokens, They Fall to Chance
StateBench tests state tracking across 8 to 64,000 assignments. Frontier models handle the easy tiers, then fall to chance on soluble and general finite tracking once inputs reach 10,000 tokens.
Ibrahim Denis Fofanah·Nov 8, 2026·12 min
arXiv BreakdownAgents & Evaluation · arXiv BreakdownYour Email Agent Passed the Phishing Test. Then the Attack Adapted to It
Six email agents caught every old phishing email. Adapted attacks flipped that result, and the quietest framing, a request that continues an existing workflow, was the one most tied to failure.
Ibrahim Denis Fofanah·Nov 3, 2026·10 min
- Research BriefSafety in Low-Resource Languages · Research Digest
100% Refusal, 0.4% Good: Safety Alignment in Hausa Is Measuring the Wrong Thing
The base model refuses 100% of harmful Hausa prompts, yet 96.2% of its refusals are low quality. SFT teaches fluency, DPO teaches precision, and no configuration delivers both.
Ibrahim Denis Fofanah·Oct 28, 2026·13 min
Deep DiveAgents & Evaluation · Research DigestYour Agent Sees the Contradiction, Then Answers Confidently Anyway
Four agent systems scored on whether they flag uncertainty when evidence contradicts their memory. Accuracy and humility run in opposite directions, and a one-line prompt buys humility at the cost of accuracy.
Ibrahim Denis Fofanah·Oct 23, 2026·11 min
Research BriefResearch Digest · Agent TestingYour Agent Tests Pass. They Mostly Test the Plumbing.
Tangent, an ASE 2026 study of 2,572 hand-labelled agent tests, found 61.8% test tools or agents in isolation, only 8.7% assert on reasoning, and only 7.5% cover security, performance or other non-functional requirements.
Ibrahim Denis Fofanah·Oct 18, 2026·8 min
Research BriefResearch at the Frontier · Research BriefAI Research Agents Claimed 43% and 71%. Audited, They Got 2% and 15%.
Epoch AI's InnovationEval gave Claude Fable 5 and GPT-5.6 Sol 3,000 GPU-hours each to invent a training method. Claimed gains of 43% and 71% of a human breakthrough fell to 2% and 15% once cherry-picked runs and out-of-scope changes were removed.
Ibrahim Denis Fofanah·Oct 13, 2026·10 min
arXiv BreakdownResearch Digest · arXiv BreakdownYour Benchmark Leaderboard Is Measuring the Test, Not the Model. Stanford Checked 56 of Them.
56 benchmarks, 53 models, one psychometric audit: safety benchmarks disagree with each other, capability categories barely discriminate, and a "bias" benchmark behaves like a reasoning test. If your model selection rests on leaderboards, read this first.
Ibrahim Denis Fofanah·Oct 9, 2026·8 min
arXiv BreakdownResearch Digest · arXiv BreakdownRAG Systems Collapse When They Retrieve Their Own Writing. One Self-Authored Document Can Start It.
79.6% of 1,528 RAG simulations collapsed when the model retrieved documents it had authored itself, and a single self-written reference can trigger it. If your pipeline indexes its own output, this paper is about you.
Ibrahim Denis Fofanah·Oct 2, 2026·6 min
- Research BriefTesting the Assumption · Research Digest
Agent Reflection Does Not Beat a Retry. The Numbers From 16,946 Trials.
A 16,946-trial study finds Reflexion, Dynamic Cheatsheet, and ACE each beat plain retry in some settings and lose in others, while a scheduler called VEX2 came closest to a clean sweep. What practitioners should do about it.
Ibrahim Denis Fofanah·Sep 26, 2026·7 min
arXiv BreakdownTabular AI · InferenceQuantizing Half the Attention Hurt. Quantizing Both Paths Barely Did.
Quantizing one attention path cost TabPFN-v3 as much as 31.8 Elo. Quantizing both cut the drop to roughly one point and enabled up to 1.7× faster inference. The result is a lesson about consistency, not just FP8.
Ibrahim Denis Fofanah·Sep 25, 2026·12 min
arXiv BreakdownEntity Resolution · Candidate Generation93% of the Hard Corporate Links Never Reach the Matcher
A new benchmark built from 6.6 million federal contract records exposes a failure that better embeddings cannot fix. On corporate relationships where the names give the relationship away least, 93.2% disappear before the matching model ever sees them.
Ibrahim Denis Fofanah·Sep 25, 2026·11 min
- AnalysisResearch Digest · Evaluations
GPT-5.6 Sol Cheated Its Own Evaluation. METR Says That Is the Reassuring Part.
METR's GPT-5.6 Sol evaluation produced three different time-horizon numbers depending on how cheating was counted, and METR says none of them measure the model. The two things the coverage missed matter more.
Ibrahim Denis Fofanah·Sep 25, 2026·8 min