Knowing–doing gap: why agents miss tool calls

Start here

In everyday words

The paper argues that some models understand that a calculator or lookup is needed, but still answer directly instead of calling the tool. That gap can make agent systems less reliable, even when the model’s reasoning seems correct.

Need a meaning?

Quick Sip

What you need to know

Who is affected: developers, researchers, operators
What changed: On May 13, 2026, researchers posted an arXiv paper introducing a model-adaptive definition of tool necessity and reporting sizable mismatches between when tools appear needed and when LLMs actually make tool-call actions on arithmetic and factual QA tasks.
Why it matters: As assistants become agents, “knowing when to use a tool” is not enough — systems must reliably translate that recognition into the right action. If models frequently fail at the last step, real-world workflows can silently degrade, making audits, evaluations, and guardrails more important than optimistic demos.
What to watch next: Watch whether agent frameworks add explicit checks for “tool necessity,” how evaluation suites measure tool-call reliability, and whether training or decoding changes can reduce last-step failures without increasing unnecessary tool calls.

Four useful details

The paper defines tool necessity relative to each model’s demonstrated capability, not a single universal label.
It reports large mismatches between tool necessity and observed tool calls across tested models and tasks.
The authors attribute many failures to a cognition-to-action transition where recognizing the need does not trigger the tool-call behavior.

arXiv · Research PaperPosition: Behavioral Systems Require Behavioral Tests

Adds source-backed context on ai research from arXiv.