Prompt Engineering in Production

1. When Prompt Engineering is Required

For everyday use: summarizing a document, brainstorming ideas or answering questions, you rarely need a rigorous engineering process. You iterate in the chat window, it works well enough, and you move on. Prompt engineering as a discipline kicks in when the stakes change.

2. From Clever Tricks to System Design

Early language models responded dramatically differently to small phrasings. Practitioners discovered "magic" patterns: "Act as a senior engineer...", "Think step by step...", "You will be penalized for..." These were real workarounds for limited model capabilities, not engineering principles.

Modern frontier models understand natural language well. The incantations matter far less. What matters now is what you put into context, not just how you phrase it. This shift from word-level tricks to context-level architecture is the defining change in the discipline.

Context Engineering

Context engineering means deliberately designing everything that enters the model's context window. For a production system, this includes:

3. Modern Prompt Engineering Techniques

System Message Design

A strong system message does three things: sets the role, defines constraints, and specifies the output format. Vague roles produce vague outputs. Explicit constraints prevent common failure modes before they happen.

import anthropic

client = anthropic.Anthropic()

system_message = """You are a meal planning assistant for competitive athletes.

CONSTRAINTS:
- Always include exact macro breakdowns (protein, fat, carbs in grams)
- Always specify meal timing (e.g., 7:00 AM, 12:00 PM)
- Never suggest foods that violate the athlete's dietary restrictions
- Always list portion sizes in grams

OUTPUT FORMAT:
- Calorie total for the day
- Macro breakdown (protein/fat/carbs)
- Numbered meal list with timing, foods, portions, and per-meal macros"""

response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    system=system_message,
    messages=[
        {"role": "user", "content": "Create a meal plan for a 75kg runner training for a marathon."}
    ]
)

Structured Input with XML Tags

Claude is trained to recognize XML-style tags as semantic separators. Using tags like <athlete_information> or <sample_input> helps the model distinguish between data, instructions, and examples - reducing ambiguity and improving consistency.

Before and After: The Meal Plan Example

The reference project includes two versions of the same meal plan prompt. The v1 version is intentionally naive. The v2 version adds three key improvements: structured XML input, numbered guidelines, and a few-shot example. The average evaluation score jumps from 4.3/10 to 8.7/10.

from evaluator.prompt_evaluator import add_user_message, chat


def run_prompt(prompt_inputs: dict) -> str:
    prompt = f"""
    Write a one-day meal plan for an athlete.

    Height: {prompt_inputs["height"]}cm
    Weight: {prompt_inputs["weight"]}kg
    Goal: {prompt_inputs["goal"]}
    Dietary restrictions: {prompt_inputs["restrictions"]}
    """
    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

from evaluator.prompt_evaluator import add_user_message, chat


def run_prompt(prompt_inputs: dict) -> str:
    prompt = f"""
    Generate a one-day meal plan for an athlete that meets their dietary restrictions.

    <athlete_information>
    - Height: {prompt_inputs["height"]}
    - Weight: {prompt_inputs["weight"]}
    - Goal: {prompt_inputs["goal"]}
    - Dietary restrictions: {prompt_inputs["restrictions"]}
    </athlete_information>

    Guidelines:
    1. Include accurate daily calorie amount
    2. Show protein, fat, and carb amounts
    3. Specify when to eat each meal
    4. Use only foods that fit restrictions
    5. List all portion sizes in grams
    6. Keep budget-friendly if mentioned

    Here is an example with a sample input and an ideal output:
    <sample_input>
    height: 170, weight: 70
    goal: Maintain fitness and improve cholesterol levels
    restrictions: High cholesterol
    </sample_input>
    <ideal_output>
    Calorie Target: approximately 2500 calories
    Macros: Protein (140g), Fat (70g), Carbs (340g)

    Breakfast (7:00 AM): Oatmeal (80g) with berries (100g) and walnuts (15g). Skim milk (240g).
      Protein: 15g, Fat: 15g, Carbs: 60g
    Mid-Morning Snack (10:00 AM): Apple (150g) with almond butter (30g).
      Protein: 7g, Fat: 18g, Carbs: 25g
    Lunch (1:00 PM): Grilled chicken breast (120g) salad with mixed greens (150g) and vinaigrette (30g).
      Protein: 40g, Fat: 15g, Carbs: 70g
    Dinner (7:00 PM): Baked salmon (140g) with steamed broccoli (200g) and quinoa (75g dry).
      Protein: 40g, Fat: 20g, Carbs: 80g
    </ideal_output>
    """
    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

4. Prompt Evaluation: The Core Discipline

Evaluation is what separates prompt engineering from prompt guessing. Without it, every change is a bet. With it, you can prove a change improved quality, quantify the gain, and catch regressions before they reach users. This is the most important skill in the discipline today.

The Evaluation Pipeline

Building an Evaluation Dataset

A good dataset covers the distribution of real inputs your system will see, including edge cases that are rare but important. For the meal plan example, this means athletes with unusual goals, strict dietary restrictions, and conflicting constraints. You can generate this dataset automatically using Claude itself: ask it to brainstorm diverse scenarios, then convert each scenario into a structured test case with inputs and success criteria.

Model-as-Judge

Manual evaluation does not scale. Instead, use a second Claude call to score each output, this is called model-as-judge. The judge receives the task description, the inputs, the output to evaluate, and the success criteria. It returns a structured score (1-10) with reasoning.

The critical detail: run the judge at temperature=0.0 to ensure deterministic, reproducible grading. The same output should always receive the same score. Using Claude Haiku for judging keeps costs low even across large datasets.

def grade_output(self, test_case: dict, output: str) -> dict:
    """Score a prompt output 1-10 using Claude as the judge."""

    eval_prompt = f"""
    Evaluate the following AI output with extreme rigor.

    Task: {test_case['task_description']}
    Inputs: {test_case['prompt_inputs']}
    Output to evaluate: {output}
    Success criteria: {test_case['solution_criteria']}

    Scoring guide:
    - Score 1-3: fails one or more mandatory requirements
    - Score 4-6: meets requirements but has significant gaps
    - Score 7-8: meets all requirements with minor issues
    - Score 9-10: fully satisfies all criteria

    Return JSON with keys: strengths, weaknesses, reasoning, score (1-10)
    """

    messages = []
    add_user_message(messages, eval_prompt)
    # temperature=0.0 ensures deterministic, reproducible grading
    result = chat(messages, temperature=0.0)
    return json.loads(result)

Metrics to Track

5. CI/CD for Prompts

A prompt is code. It should be versioned, reviewed, and tested before it ships. The same rigor that prevents a bad function from reaching production should prevent a broken prompt from reaching users. In practice this means running your evaluation suite automatically on every prompt change, and blocking the merge if scores regress beyond a defined threshold.

The Automated Evaluation Flow

Regression Prevention

A common pattern: define a minimum acceptable average score (e.g., 7.5/10) and a minimum pass rate (e.g., 80%). A prompt change that drops either metric below the threshold fails the CI check. This enforces the rule that you can only ship a prompt change if you can prove it did not make things worse.

The compare-prompts command in the reference project demonstrates this: it runs both prompt versions against the same dataset and prints the score delta side-by-side.

6. Reference Project: Meal Plan Evaluator

The reference project at gitlab.com/bytecode-solutions/examples/prompt-engineering implements the full evaluation workflow using the Anthropic Python SDK. It is structured as a reusable evaluator/ package with a concrete meal plan example in examples/meal_plan/.

Three-Step Workflow

# Step 1: Generate a diverse dataset of test cases using Claude
python manager.py generate-dataset --num-cases 10

# Step 2: Run the improved prompt against the dataset and generate an HTML report
python manager.py run-evaluation

# Step 3: Compare the naive prompt (v1) against the improved prompt (v2) side-by-side
python manager.py compare-prompts

Expected Output

# (example output, actual scores vary by dataset and model run)

Evaluating v1: naive  (no structure, no few-shot)...
  Average score: 4.3/10 --> examples/meal_plan/output_v1.html

Evaluating v2: improved (structured guidelines + few-shot example)...
  Average score: 8.7/10 --> examples/meal_plan/output_v2.html

--- Summary ---
  v1 (naive):    4.3/10
  v2 (improved): 8.7/10
  Improvement:   +4.4 points (+102%)

Open the HTML reports to understand why scores differ:
  v1: examples/meal_plan/output_v1.html
  v2: examples/meal_plan/output_v2.html