LLM cost control for teams: budgets, routing, and guardrails

LLM bills surprise teams that treat chat like free coffee. Cost control is an ops problem: measure tokens, set budgets, route work to cheaper models when quality allows, and stop paying for retries and redundant context. Use the token estimator to forecast before you ship a feature.

What actually drives spend

1. Input tokens — system prompts, retrieval dumps, chat history, tool schemas.
2. Output tokens — long answers, JSON with comments, verbose “thinking” style.
3. Call volume — users, cron jobs, agents looping, eval suites.
4. Retries and fallbacks — timeouts, rate limits, “try again with more context.”
5. Model tier — frontier models cost multiples of mid-tier or small models.

A 2,000-token system prompt × 100k calls/month is 200M input tokens before any user text. That alone can dominate the invoice.

Step 1 — Instrument before you optimize

Log per request (or sample 10%):

• model id
• input / output tokens (from the API usage object)
• feature or route name
• user or tenant id (hashed if needed)
• latency and retry count
• cache hit / miss if applicable

Weekly rollup: cost by feature, by model, by tenant. Without this, “we’re over budget” is a guess.

Step 2 — Set hard budgets

Layer	Example
Org monthly	Soft alert at 70%, hard stop non-critical jobs at 100%
Feature	Content draft API: $X/day
Tenant	Free tier: N tokens/day
Agent loop	Max M tool calls or $Y per task

Hard stops need a clear UX: queue gracefully, queue to a cheaper model, or queue for human approval — do not silently drop user work.

Step 3 — Model routing matrix

Not every task needs the strongest model.

Task	Prefer	Escalate when
Classification, tagging	Small / cheap	Low confidence
Outline, rewrite for clarity	Mid-tier	Legal / brand risk
Complex reasoning, long synthesis	Frontier	—
Code generation with tests	Mid + verify	Tests fail twice
Customer-facing sensitive copy	Mid + human edit	Always human for claims

Encode the matrix in config, not tribal knowledge. Revisit monthly as prices and models change.

Step 4 — Shrink input without starving quality

• Trim system prompts. Move essays into docs; keep constraints short. See production system prompts.
• Cap history. Summarize older turns; do not resend 40 messages.
• Retrieve less. Top-k = 3–5 relevant chunks beats dumping the wiki.
• Dedupe context. Same style sheet pasted into every call? Cache or reference an ID.
• Strip noise. Clean pasted notes with the prompt cleaner.

Step 5 — Control output length

• Set max_tokens deliberately (e.g. 400 for titles/meta, 1,200 for a section).
• Ask for bullets or tables when prose is unnecessary.
• Ban “restating the question” in system rules.
• For JSON APIs, use schemas and reject padded commentary.

Step 6 — Caching and batching

• Prompt / prefix caching (where the provider supports it): stable system + tools first; volatile user text last.
• Semantic cache for identical FAQ answers (careful with personalization and staleness).
• Batch offline jobs (evals, bulk rewrites) during cheaper windows if your vendor offers batch pricing.
• Avoid N+1 agent calls — plan tool use, then execute once.

Step 7 — Retry policy that does not burn money

• Exponential backoff with a max retry count (e.g. 2).
• Do not retry with a larger model automatically unless quality metrics demand it.
• On timeout, prefer idempotent short requests over “send the whole conversation again.”
• Separate user-visible errors from background eval retries so product UX does not amplify spend.

Monthly forecasting worksheet

For each feature:

monthly_cost ≈ calls × (1 + retry_rate) × (
  (avg_in_tokens × in_price_per_M / 1e6) +
  (avg_out_tokens × out_price_per_M / 1e6)
)

Keep an editable rate table (the site estimator does this locally). Re-forecast when you change system prompts or retrieval size — those shifts often matter more than traffic spikes.

Governance that teams actually follow

1. Owner per high-spend route — a named engineer, not “the AI squad.”
2. Change review for system prompt or context window increases >20%.
3. Eval cost budget — nightly suites can outspend production if unchecked.
4. Quarterly vendor review — compare unit economics across providers; see also token counting explained.

Anti-patterns

• Unlimited agent loops “until done”
• Logging full prompts to a paid analytics LLM
• Using frontier models for spam classification
• Shipping a 5k-token “personality” preamble
• Ignoring free-tier abuse (bots hitting your chat UI)

30-day cost control sprint

Week 1: instrument + baseline. Week 2: routing matrix + max_tokens. Week 3: trim top 3 prompts by input size. Week 4: budgets + alerts. Expect the biggest wins from input trim and routing, not from shaving 5% off output.

For solopreneurs assembling a lean stack, see AI tool stack for solopreneurs. For writing-model tradeoffs that affect both quality and price, read ChatGPT vs Claude vs Gemini for writing.