Prompt Engineering in Production

Introduction

Lesson 3 covered how to write effective prompts: roles, few-shot examples, chain-of-thought. This lesson is about the next level, using prompts in production systems where reliability, repeatability, and measurability are non-negotiable. The craft has evolved from a standalone job into a software engineering discipline: you need evaluation datasets, metrics, regression prevention, and automated pipelines to confidently change a prompt and ship it to users.

1. When Prompt Engineering is Required

For everyday use: summarizing a document, brainstorming ideas or answering questions, you rarely need a rigorous engineering process. You iterate in the chat window, it works well enough, and you move on. Prompt engineering as a discipline kicks in when the stakes change.

2. From Clever Tricks to System Design

Early language models responded dramatically differently to small phrasings. Practitioners discovered "magic" patterns: "Act as a senior engineer...", "Think step by step...", "You will be penalized for..." These were real workarounds for limited model capabilities, not engineering principles.

Modern frontier models understand natural language well. The incantations matter far less. What matters now is what you put into context, not just how you phrase it. This shift from word-level tricks to context-level architecture is the defining change in the discipline.

Context Engineering

Context engineering means deliberately designing everything that enters the model's context window. For a production system, this includes:

3. Modern Prompt Engineering Techniques

System Message Design

A strong system message does three things: sets the role, defines constraints, and specifies the output format. Vague roles produce vague outputs. Explicit constraints prevent common failure modes before they happen.

# system_message.py

import anthropic

client = anthropic.Anthropic()

system_message = """You are a meal planning assistant for competitive athletes.

CONSTRAINTS:
- Always include exact macro breakdowns (protein, fat, carbs in grams)
- Always specify meal timing (e.g., 7:00 AM, 12:00 PM)
- Never suggest foods that violate the athlete's dietary restrictions
- Always list portion sizes in grams

OUTPUT FORMAT:
- Calorie total for the day
- Macro breakdown (protein/fat/carbs)
- Numbered meal list with timing, foods, portions, and per-meal macros"""

response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    system=system_message,
    messages=[
        {"role": "user", "content": "Create a meal plan for a 75kg runner training for a marathon."}
    ]
)

print(response.content[0].text[:280])

# Marathon Training Meal Plan for 75kg Runner

**Daily Totals:**
- **Calories:** 2,850 kcal
- **Protein:** 135g (19%)
- **Fat:** 85g (27%)
- **Carbs:** 355g (50%)

---

## Daily Meal Schedule

### **MEAL 1: 6:30 AM (Pre-Dawn Run)**
**Timing:** 30-45 min before 10-12km easy run

-

Structured Input with XML Tags

Claude is trained to recognize XML-style tags as semantic separators. Using tags like <athlete_information> or <sample_input> helps the model distinguish between data, instructions, and examples - reducing ambiguity and improving consistency.

Before and After: The Meal Plan Example

The reference project includes two versions of the same meal plan prompt. The v1 version is intentionally naive. The v2 version adds three improvements: structured XML input, numbered guidelines, and a few-shot example. Section 6 measures both against the same dataset, and the result is worth seeing before you assume structure always wins: on claude-haiku-4-5 the two versions score within noise of each other.

# examples/meal_plan/prompt_v1.py

from evaluator.prompt_evaluator import add_user_message, chat


def run_prompt(prompt_inputs: dict) -> str:
    """Send a minimal, unguided meal plan request to the model."""

    prompt = f"""
    Write a one-day meal plan for an athlete.

    Height: {prompt_inputs["height"]}cm
    Weight: {prompt_inputs["weight"]}kg
    Goal: {prompt_inputs["goal"]}
    Dietary restrictions: {prompt_inputs["restrictions"]}
    """

    messages: list = []
    add_user_message(messages, prompt)
    return chat(messages)

# examples/meal_plan/prompt.py

from evaluator.prompt_evaluator import add_user_message, chat


def run_prompt(prompt_inputs: dict) -> str:
    """Build and send the meal plan prompt; return the raw model response."""
    prompt = f"""
    Generate a one-day meal plan for an athlete that meets their dietary restrictions.

    <athlete_information>
    - Height: {prompt_inputs["height"]}
    - Weight: {prompt_inputs["weight"]}
    - Goal: {prompt_inputs["goal"]}
    - Dietary restrictions: {prompt_inputs["restrictions"]}
    </athlete_information>

    Guidelines:
    1. Include accurate daily calorie amount
    2. Show protein, fat, and carb amounts
    3. Specify when to eat each meal
    4. Use only foods that fit restrictions
    5. List all portion sizes in grams
    6. Keep budget-friendly if mentioned

    Here is an example with a sample input and an ideal output:
    <sample_input>
    height: 170
    weight: 70
    goal: Maintain fitness and improve cholesterol levels
    restrictions: High cholesterol
    </sample_input>
    <ideal_output>
    Here is a one-day meal plan for an athlete aiming to maintain fitness and improve cholesterol levels:

    *   **Calorie Target:** Approximately 2500 calories
    *   **Macronutrient Breakdown:** Protein (140g), Fat (70g), Carbs (340g)

    **Meal Plan:**

    *   **Breakfast (7:00 AM):** Oatmeal (80g dry weight) with berries (100g) and walnuts (15g). Skim milk (240g).
        *   Protein: 15g, Fat: 15g, Carbs: 60g
    *   **Mid-Morning Snack (10:00 AM):** Apple (150g) with almond butter (30g).
        *   Protein: 7g, Fat: 18g, Carbs: 25g
    *   **Lunch (1:00 PM):** Grilled chicken breast (120g) salad with mixed greens (150g), cucumber (50g), tomato (50g), and a light vinaigrette dressing (30g). Whole wheat bread (60g).
        *   Protein: 40g, Fat: 15g, Carbs: 70g
    *   **Afternoon Snack (4:00 PM):** Greek yogurt (170g, non-fat) with a banana (120g).
        *   Protein: 20g, Fat: 0g, Carbs: 40g
    *   **Dinner (7:00 PM):** Baked salmon (140g) with steamed broccoli (200g) and quinoa (75g dry weight).
        *   Protein: 40g, Fat: 20g, Carbs: 80g
    *   **Evening Snack (9:00 PM):** Small handful of almonds (20g).
        *   Protein: 8g, Fat: 12g, Carbs: 15g

    This meal plan prioritizes lean protein sources, whole grains, fruits, and vegetables, while limiting saturated and trans fats to support healthy cholesterol levels.
    </ideal_output>
    This example meal plan is well-structured, provides detailed information on food choices and quantities, and aligns with the athlete's goals and restrictions.
    """

    messages: list = []
    add_user_message(messages, prompt)
    return chat(messages)

4. Prompt Evaluation: The Core Discipline

Evaluation is what separates prompt engineering from prompt guessing. Without it, every change is a bet. With it, you can prove a change improved quality, quantify the gain, and catch regressions before they reach users. This is the most important skill in the discipline today.

The Evaluation Pipeline

Building an Evaluation Dataset

A good dataset covers the distribution of real inputs your system will see, including edge cases that are rare but important. For the meal plan example, this means athletes with unusual goals, strict dietary restrictions, and conflicting constraints. You can generate this dataset automatically using Claude itself: ask it to brainstorm diverse scenarios, then convert each scenario into a structured test case with inputs and success criteria.

Model-as-Judge

Manual evaluation does not scale. Instead, use a second Claude call to score each output, this is called model-as-judge. The judge receives the task description, the inputs, the output to evaluate, and the success criteria. It returns a structured score (1-10) with reasoning. Using Claude Haiku for judging keeps costs low even across large datasets.

Two details make the difference between a judge that works and one that crashes on the first call. Set temperature=0.0 to narrow the spread of scores. Then force the reply to be parseable: prefill the assistant turn with an opening code fence and stop on the closing one. Without that, the model wraps its JSON in a Markdown fence and json.loads raisesJSONDecodeError on the very first character.

# grade_output.py

import json

from anthropic import Anthropic

client = Anthropic()
JUDGE_MODEL = "claude-haiku-4-5"


def grade_output(test_case: dict, output: str, extra_criteria: str | None = None) -> dict:
    """Score a prompt output 1-10 using Claude as the judge."""

    extra_section = ""
    if extra_criteria:
        extra_section = f"""
    Mandatory requirements, ANY violation caps the score at 3:
    <extra_criteria>
    {extra_criteria}
    </extra_criteria>
    """

    eval_prompt = f"""
    Evaluate the following AI output with EXTREME RIGOR.

    <task_description>{test_case["task_description"]}</task_description>
    <task_inputs>{json.dumps(test_case["prompt_inputs"])}</task_inputs>
    <solution>{output}</solution>
    <criteria>{chr(10).join(test_case["solution_criteria"])}</criteria>
    {extra_section}

    Scoring guidelines:
    - Score 1-3: fails one or more mandatory requirements
    - Score 4-6: meets mandatory requirements but has significant gaps
    - Score 7-8: meets all requirements with minor issues
    - Score 9-10: fully satisfies all criteria

    Reply with JSON only, keys in this order:
    "strengths" (string[]), "weaknesses" (string[]), "reasoning" (string), "score" (number).
    """

    response = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=1000,
        temperature=0.0,
        messages=[
            {"role": "user", "content": eval_prompt},
            # Prefilling the assistant turn commits the reply to raw JSON, and
            # the stop sequence ends it before the closing fence. Without this,
            # the model wraps the object in ```json and json.loads raises.
            {"role": "assistant", "content": "```json"},
        ],
        stop_sequences=["```"],
    )
    return json.loads(response.content[0].text)


if __name__ == "__main__":
    case = {
        "task_description": "Write a one-day meal plan for an athlete.",
        "prompt_inputs": {"height": "180", "weight": "75",
                          "goal": "Build lean muscle", "restrictions": "Lactose intolerant"},
        "solution_criteria": ["States a daily calorie target",
                              "Gives protein, fat and carb amounts in grams",
                              "Specifies a time for each meal",
                              "Lists portion sizes in grams"],
    }
    weak_output = ("Breakfast: oats. Lunch: chicken and rice. Dinner: salmon and "
                   "vegetables. Eat enough protein to support training.")

    verdict = grade_output(case, weak_output, extra_criteria="Must avoid all dairy")
    print(f"score      : {verdict['score']}")
    print(f"weaknesses : {verdict['weaknesses'][0]}")
    print(f"reasoning  : {verdict['reasoning'][:96]}")

score      : 4
weaknesses : Does not state a daily calorie target
reasoning  : The solution meets the mandatory requirement by avoiding all dairy products, preventing a score 

What temperature=0.0 Does Not Buy You

It is tempting to describe a judge at temperature 0 as deterministic. It is not. Temperature 0 makes token sampling greedy, but it does not make the whole serving path reproducible, and Anthropic does not document it as deterministic. Grade one fixed output eight times and watch:

# variance_check.py

import hashlib
from collections import Counter

from grade_output import grade_output

CASE = {
    "task_description": "Write a one-day meal plan for an athlete.",
    "prompt_inputs": {"height": "180", "weight": "75",
                      "goal": "Build lean muscle", "restrictions": "Lactose intolerant"},
    "solution_criteria": ["States a daily calorie target",
                          "Gives protein, fat and carb amounts in grams",
                          "Specifies a time for each meal",
                          "Lists portion sizes in grams"],
}

# A deliberately borderline answer: complete enough to argue about.
BORDERLINE = """Daily target: 2800 kcal. Protein 165g, Fat 80g, Carbs 330g.
Breakfast (7:00 AM): oats (80g) with almond milk (240g) and blueberries (100g).
Lunch (12:30 PM): chicken breast (150g), rice (90g dry), broccoli (150g).
Dinner (7:00 PM): salmon (160g), sweet potato (200g), spinach (100g).
Snack: almonds (30g) and a banana."""

scores, digests = [], []
for _ in range(8):
    verdict = grade_output(CASE, BORDERLINE, extra_criteria="Must avoid all dairy")
    scores.append(verdict["score"])
    digests.append(hashlib.sha256(verdict["reasoning"].encode()).hexdigest()[:8])

print(f"scores over 8 identical calls : {scores}")
print(f"score distribution            : {dict(Counter(scores))}")
print(f"distinct reasoning texts      : {len(set(digests))} of 8")
print(f"same score every time         : {len(set(scores)) == 1}")

scores over 8 identical calls : [9, 7, 7, 8, 8, 8, 9, 8]
score distribution            : {9: 2, 7: 2, 8: 4}
distinct reasoning texts      : 7 of 8
same score every time         : False

Eight identical calls, three different scores, and seven different pieces of reasoning. Run it again and you will get a different mix: that is the point, and it is why this block is the one place in the lesson where your numbers are expected not to match. The spread straddles the 7-and-above line that the pass rate below is built on, so the same answer counts as a pass on one run and a failure on the next. Clear-cut cases are stable, borderline ones are not, and borderline cases are exactly the ones a threshold decides.

This does not make evaluation useless, it makes single-run comparisons useless. Treat a score as a measurement with error bars: average across the dataset, compare distributions rather than points, and require a margin larger than the noise before calling a prompt change an improvement.

Metrics to Track

5. CI/CD for Prompts

A prompt is code. It should be versioned, reviewed, and tested before it ships. The same rigor that prevents a bad function from reaching production should prevent a broken prompt from reaching users. In practice this means running your evaluation suite automatically on every prompt change, and blocking the merge if scores regress beyond a defined threshold.

The Automated Evaluation Flow

Regression Prevention

A common pattern: define a minimum acceptable average score (e.g., 7.5/10) and a minimum pass rate (e.g., 80%). A prompt change that drops either metric below the threshold fails the CI check. This enforces the rule that you can only ship a prompt change if you can prove it did not make things worse.

A hard threshold on a noisy metric fails in both directions, and the noise is not small. Running the reference project's comparison three times against one unchanged dataset gave the naive prompt 8.1, then 7.5, then 7.4. A gate set at 7.5 would have passed that prompt twice and failed it once with nothing changed but the run. Before choosing a threshold, run your suite several times on a frozen dataset, measure the spread, and set the gate outside it. Comparing two prompts on the same dataset in the same run, as compare-prompts does, cancels much of the drift that comparing against last week's number does not.

The compare-prompts command in the reference project demonstrates this: it runs both prompt versions against the same dataset and prints the score delta side-by-side.

6. Reference Project: Meal Plan Evaluator

The reference project at gitlab.com/bytecode-solutions/examples/prompt-engineering implements the full evaluation workflow using the Anthropic Python SDK. It is structured as a reusable evaluator/ package with a concrete meal plan example in examples/meal_plan/.

Three-Step Workflow

# Run from the project root

# Step 1: Generate a diverse dataset of test cases using Claude
python manager.py generate-dataset --num-cases 10

# Step 2: Run the improved prompt against the dataset and generate an HTML report
python manager.py run-evaluation

# Step 3: Compare the naive prompt (v1) against the improved prompt (v2) side-by-side
python manager.py compare-prompts

What the Comparison Actually Reports

Here is one real run of compare-prompts over a freshly generated 10-case dataset, judged by claude-haiku-4-5:

# python manager.py compare-prompts --concurrency 5

Evaluating v1: naive  (no structure, no few-shot)...
Graded 2/10 test cases
Graded 4/10 test cases
Graded 6/10 test cases
Graded 8/10 test cases
Graded 10/10 test cases
Average score: 8.1
  Average score: 8.1/10 --> examples/meal_plan/output_v1.html

Evaluating v2: improved (structured guidelines + few-shot example)...
Graded 2/10 test cases
Graded 4/10 test cases
Graded 6/10 test cases
Graded 8/10 test cases
Graded 10/10 test cases
Average score: 7.6
  Average score: 7.6/10 --> examples/meal_plan/output_v2.html

--- Summary ---
  v1 (naive):    8.1/10
  v2 (improved): 7.6/10
  Improvement:   -0.5 points (-6%)

Open the HTML reports to understand why scores differ:
  v1: examples/meal_plan/output_v1.html
  v2: examples/meal_plan/output_v2.html

The carefully structured prompt lost. Repeating the same command twice more on that same dataset gave a different answer each time: