How to estimate LLM token costs before you ship a feature

Token bills surprise teams that ship first and measure later. You do not need perfect telemetry on day one — you need a spreadsheet-grade estimate that is honest about retries and context growth.

The cost formula that actually matters

For a single call:

cost ≈ (input_tokens × input_rate) + (output_tokens × output_rate)

Rates are usually listed per 1M tokens. Convert early so you do not misplace zeros.

For a feature:

monthly_cost ≈ calls_per_month × cost_per_call × (1 + retry_rate) × overhead

overhead covers system prompts, tool schemas, and RAG chunks that you forget to count.

Step 1 — Measure a real input

Take 10–20 production-shaped examples (not the happy toy prompt). For each:

1. Count characters or use a tokenizer approximation (~4 chars/token for English prose; code is denser).
2. Add the fixed system prompt + tool definitions.
3. Add retrieved context if you use RAG (this is where budgets explode).

Average those sizes. Prefer p90 over mean if traffic is skewed.

Step 2 — Bound the output

Cap max_tokens in the API. Unbounded generation is a cost and latency bug. Estimate average completion length from a dry run of 20 calls.

Step 3 — Price the model you will actually use

Vendor list prices change. Keep an editable rates table (the token estimator on this site is built for that). Track:

Model tier	Input / 1M	Output / 1M	Notes
Small / fast	$X	$Y	Classification, formatting
Mid	$X	$Y	Default app traffic
Large	$X	$Y	Hard reasoning only

Route by task. Paying large-model rates for “normalize this JSON” is how bills grow.

Step 4 — Multiply by real traffic patterns

Include:

• Retries on timeouts and malformed JSON (often 5–20%)
• Agent loops (N tool calls × N reasoning turns)
• Human re-rolls in chat UIs
• Eval suites run in CI

A feature that looks like $0.002/call becomes material at 2M calls/month with a 15% retry rate and a 3-turn agent.

Step 5 — Cut cost without killing quality

Practical levers, in order of ROI:

1. Shrink context — drop unused tools, summarize history, retrieve fewer chunks.
2. Cache stable prefixes when the provider supports it.
3. Batch offline jobs; do not pay interactive latency prices for nightly work.
4. Cascade models — classify with a small model, escalate only when unsure.
5. Validate in code — do not ask the model to re-explain what a schema validator can catch.

Worked sketch

Assume:

• 500k calls/month
• 1,200 input tokens (p90), 300 output tokens
• Mid model: $0.50 / 1M in, $1.50 / 1M out
• 10% retries

per_call = (1200/1e6)*0.50 + (300/1e6)*1.50 = $0.00105
monthly  = 500000 * 0.00105 * 1.10 ≈ $577

If agents average 4 turns, multiply again. That is the conversation you want with product before launch — not after the invoice.

Checklist before you flip the feature flag

• p90 input size measured on real data
• max_tokens set
• Model tier chosen per task class
• Retries and agent turns included
• Alert on spend / day
• Rates table owned by someone (and editable)

Estimate early. Revisit when prompts or retrieval change — those are silent cost regressions.