Before you ship that LLM feature: five checks that save you a 2 a.m. rollback

The demo always goes well. You picked ten good inputs, the model did something impressive, and everyone nodded. Then real users show up with empty fields, emoji, three languages in one message and a PDF that turns out to be a scanned image.

None of the checks below are clever. They are the boring work that decides whether the feature survives its first week.

1. Structured output, validated every time

Free text is fine for a chat window. Anywhere your code reads the result, ask for structured output and validate it against a schema before using it. Most providers now support JSON schema or tool calling for this.

from pydantic import BaseModel, ValidationError

class Triage(BaseModel):
    category: str
    urgency: int  # 1 to 3
    reply_draft: str

def parse(raw: str) -> Triage | None:
    try:
        return Triage.model_validate_json(raw)
    except ValidationError:
        return None  # count it, retry once, then fall back

If validation fails, retry once with the error message included, then fall back. Never let a half-parsed answer flow into your database.

2. An evaluation set built from real data

Collect 50 to 200 real inputs (anonymised), including the ugly ones: empty, very long, wrong language, sarcastic, irrelevant. For each, write down what a good output looks like, or at least what a bad one looks like.

Run that set on every prompt or model change, in CI if you can. You are not looking for a perfect score; you are looking for "did this change make things worse". Without it, every prompt tweak is a guess.

3. Budgets: tokens, time and money

• Set a maximum output length for every call.
• Set a timeout and decide what the user sees when it hits.
• Estimate cost per request and per user per day, and alert when real numbers drift above it.
• Cache answers for identical inputs where it is safe.

A loop that accidentally calls the model a thousand times is a bad way to learn about your bill.

4. A plan for when the provider fails

Providers have outages, rate limits and slow days. Decide in advance:

• Retry with backoff for rate limits, but not forever.
• Fall back to a second model or provider for critical paths, using the same eval set to check it is good enough.
• Show something useful when all else fails: "We could not draft a reply right now" plus the manual option, not a spinner that never ends.

5. Logs you can actually debug with

Log the input, output, model, prompt version, latency, token count and validation result for every call. Redact personal data before it is stored. Give prompts version numbers, and store them in your repository like code.

When someone reports a weird answer, you want to find that exact call in a minute, see which prompt produced it, and add it to your eval set so it never comes back.

A quick pre-launch list

1. Output is validated against a schema.
2. Eval set exists and runs on every change.
3. Token, time and cost limits are set, with alerts.
4. Failure path is designed and tested by switching the provider off.
5. Calls are logged with prompt versions and personal data redacted.
6. Someone owns the feature after launch.

If you are missing two or more, it is not ready. That is fine; it is a lot cheaper to find out now.

Questions people ask

Do I need a framework like LangChain or LangGraph?

Not for a single call with structured output. Frameworks help when you have multi-step agents with state, tools and human approval. Start with the provider's SDK and add a framework when the plain code starts to hurt.

How big should the eval set be?

Big enough to include your ugly cases. Fifty well-chosen examples beat five hundred similar ones. Add every bug report to it.

Should I fine-tune instead?

Usually not first. Better instructions, examples in the prompt and retrieval over your own documents solve most problems more cheaply. Fine-tune when you have a stable task, lots of good examples and evidence that prompting has hit its limit.