AI agent tool-calling token overhead

Agent demos look cheap until tool schemas and transcripts show up on the invoice. Tool-calling token overhead is the extra input (and sometimes output) you pay because the model must see tool definitions, prior calls, results, and scaffolding on every step. This guide shows how to estimate that overhead without fake “agent autonomy” marketing.

Related: RAG context cost, retry budget, token estimator, unit economics.

Where the tokens go

A single user message rarely equals a single billed prompt. Typical agent loop:

1. System prompt — policies, style, safety
2. Tool catalog — JSON/YAML schemas or API descriptions for every enabled tool
3. User message
4. Assistant tool_call — name + arguments
5. Tool result — raw or summarized
6. Repeat 4–5 until final answer

Steps 2, 4, and 5 are overhead relative to “just chat.” On long runs, prior tool results accumulate like an ever-growing RAG pack.

Measure overhead with a simple identity

For one completed agent task:

total_input_tokens ≈ sum over steps(system + tools + messages_so_far)
overhead ≈ total_input_tokens - tokens(user_visible_dialog)

More useful ratios:

• Schema tax = tokens(tool catalog) / tokens(user message)
• Transcript tax = tokens(prior tool results) / tokens(user message)
• Steps = number of model calls until final answer

Log these three in staging. They explain more cost variance than model list price.

Schema tax: shrink the catalog

Tool catalogs are often pasted wholesale.

Tactics:

• Allowlist per feature — support bot does not need deploy_k8s
• Split tools — few focused tools beat one mega-tool with 40 parameters
• Short descriptions — models need clear names and enums; not novels
• Defer rare tools — load secondary tools only after intent classification (second call, but smaller average catalog)

Count the catalog alone in a local tokenizer or the token estimator. If schema tax > 5–10× a typical user utterance, fix schemas before tuning temperature.

Transcript tax: summarize and forget

Every tool result you keep in the messages array is paid again on the next step.

Tactics:

• Summarize large tool outputs (search pages, DB dumps) before re-injecting
• Hard-cap result size in the tool adapter
• Drop stale tools from context when the plan moves on
• Prefer idempotent tools so you can omit redundant history
• Store full traces in your logs/DB, not necessarily in the model context

Treat transcript growth like RAG context packing: p90 length matters more than the happy path.

Step count: the silent multiplier

If each step re-sends ~3K input tokens and you average 6 steps, you already think in 18K input tokens before the final answer. Retries multiply again — see retry budget spreadsheet.

Design levers:

• Planner that emits a structured plan once, then executes with smaller prompts
• Deterministic code for steps that do not need a model
• Early stop when confidence or schema validation passes
• Max-step circuit breaker with user-visible degrade

Estimation worksheet

Field	Value
Tokens(system)
Tokens(tool catalog)
Tokens(avg user)
Tokens(avg tool result)
Avg steps / task
Fraction of steps that include full catalog
Retry multiplier
Output tokens / task

Rough input total:

steps * (system + catalog_or_subset + dialog_so_far_avg)

Start with a pessimistic dialog_so_far_avg (grows with steps). Then calibrate on 20 real traces.

Plug totals into the token estimator with your rates. Present $/successful task, not $/step.

Product implications

• Price plans around tasks, not raw tokens, if customers cannot see overhead.
• Show internal dashboards for schema tax and step count by feature flag.
• Cap tools in customer-facing agents more aggressively than internal research agents.
• Document overhead assumptions next to system prompt versions.

Anti-patterns

• Enabling every plugin “just in case”
• Returning entire HTML pages as tool results
• No max-step limit
• Retrying failed tool calls without removing huge error payloads from context
• Comparing agent vendors on chat-only token prices

Practical next steps

1. Export 20 production traces; compute schema tax, transcript tax, steps.
2. Cut the catalog until schema tax drops materially.
3. Add result size caps in tool adapters.
4. Re-forecast with the estimator and update unit economics.

Agent quality is a product problem. Agent cost is often a context hygiene problem wearing a trench coat.

Observability fields to log

Per agent run, log (with PII controls):

• feature_id, prompt_version, model_id
• step_index, tools_enabled_hash
• tokens_in, tokens_out (from vendor usage when available)
• tool_result_chars before and after truncation
• stop_reason (final / max_steps / error)

Dashboards: p50/p90 steps, p90 input tokens per step, $ per successful task. Without these, overhead debates stay anecdotal.