System prompts that hold up in production

A system prompt is part of your product contract. Treat it like configuration that can break users — because it can.

What belongs in the system prompt

Put durable rules here:

• Role and scope (“you are a support assistant for Product X; you do not invent pricing”)
• Output contracts (JSON schema, markdown sections)
• Safety and refusal policy specific to your domain
• Tool-use rules (when to call, when to ask the user)
• Tone constraints that rarely change

Keep volatile task details in the user message or a retrieved brief. Bloated system prompts raise cost on every call and make diffs scary.

Write it as a contract

Prefer normative language:

You MUST respond with valid JSON matching SCHEMA.
You MUST NOT invent invoice IDs.
If the user asks for legal advice, refuse and suggest contacting counsel.

Avoid soft mush: “Try to be helpful and mostly stick to JSON when possible.”

Tool-use section

If the model can call tools, spell out:

1. Allowed tools and one-line purpose each
2. When not to call (answer from provided context first)
3. How to handle tool errors (retry once vs surface error)
4. Parallel vs sequential calling policy

Ambiguous tool policy → wasted calls → latency and cost.

Versioning and rollout

1. Store prompts in git (prompts/support-v4.txt or a typed config module)
2. Change with a PR description: behavior delta + eval notes
3. Shadow-test on logged traces before 100% traffic
4. Keep a kill switch to pin the previous version

Never edit production prompts only in a vendor dashboard with no history.

Eval the prompt, not your gut

Maintain a small golden set (20–50 cases):

• Happy paths
• Missing fields
• Jailbreak-ish asks relevant to your product
• Multilingual if you support it
• Long context / noisy retrieval

Score with assertions (schema valid, required keys present, banned phrases absent). LLM-as-judge can supplement; it should not be the only gate for safety-critical behavior.

Cost and latency notes

Every token in the system prompt is paid on every request. Periodically:

• Delete dead rules
• Move rarely needed policy into conditional prefixes
• Prefer references (“follow POLICY.md section 2”) only if that text is actually in context — dangling references confuse models

Minimal template

# Role
…

# Scope & refusals
…

# Output contract
…

# Tools
…

# Style
… (short)

# Missing information
Ask at most N clarifying questions OR fill with nulls per schema.

Pair this with the prompt cleaner when consolidating notes into a tight system message, and estimate tokens before you add another essay to the prefix.