Evaluate LLM Features Before Tuning the Prompt

Prompt changes feel productive because the feedback is immediate. The problem is that an improvement on one example can quietly make a hundred other cases worse. LLM features need the same discipline as other production behavior: a representative test set, measurable expectations and regression checks.

#Build a Small Golden Set

Start with real, anonymized examples and include difficult cases:

• incomplete or ambiguous questions
• requests outside the system's knowledge
• permission-sensitive questions
• long inputs and exact identifiers
• tool failures and timeouts
• prompt-injection attempts in retrieved content

Each case should record the expected behavior, not necessarily one exact string. For a support answer, that may mean citing the correct policy, refusing an unsupported refund and asking one clarifying question.

#Test Each Layer Separately

An end-to-end score hides the cause of a failure. Test the stages independently:

retrieval -> tool selection -> argument validation -> answer quality -> safety

For retrieval, measure whether the expected source appears. For tool use, check that the right tool and arguments are selected. For answers, check factual claims, citations and refusal behavior.

#Prefer Deterministic Checks Where Possible

Use ordinary assertions for JSON schemas, tool names, permission filters, citation URLs and limits. Use model-based grading only for qualities that cannot be checked reliably with code, and keep the grader prompt versioned.

Every production incident should become a regression case. This turns an embarrassing failure into a permanent improvement.

#Track Cost and Latency

Quality is only one part of the feature. Record model, prompt version, token usage, tool calls, latency and failure reason for every evaluation run. A slightly better answer that costs ten times more may not be a better product.

LLM evaluation is not about proving that a model is intelligent. It is about proving that a specific system behaves acceptably for the work it is allowed to perform.