Evaluating & Iterating on Prompts

Writing prompts is iterative. Systematic evaluation helps you move from "it seems to work" to "it reliably produces quality results."

Why Evaluation Matters

Without systematic evaluation:

You rely on gut feeling ("this looks good")
Edge cases slip through
Quality degrades when you change models
You can't compare approaches objectively

With evaluation:

You have measurable quality benchmarks
You catch regressions early
You can A/B test different prompts
You build reliable production systems

Manual Evaluation Criteria

The 5 Dimensions of Prompt Quality

Dimension	Question	Score
Accuracy	Is the information correct?	1-5
Relevance	Does it answer what was asked?	1-5
Completeness	Is anything missing?	1-5
Format	Does it match the requested structure?	1-5
Consistency	Does it give similar results each time?	1-5

Evaluation Rubric Example

For a code generation prompt:

5/5 - Code runs correctly, handles edge cases, follows best practices
4/5 - Code runs correctly, minor style issues
3/5 - Code runs with minor bugs, needs small fixes
2/5 - Code has logical errors, partially works
1/5 - Code doesn't run or is completely wrong

Building a Test Suite

Create a set of test cases to evaluate your prompt consistently:

python

test_cases = [
    {
        "input": "Sort a list of numbers",
        "expected_contains": ["def sort", "return"],
        "expected_not_contains": ["import os", "print("],
        "language": "python"
    },
    {
        "input": "Sort an empty list",
        "expected_behavior": "Should handle gracefully, return []",
    },
    {
        "input": "Sort a list with one element",
        "expected_behavior": "Should return the same list",
    },
    {
        "input": "Sort a list with duplicate elements",
        "expected_behavior": "Should preserve all elements",
    }
]

Types of Test Cases

Happy path: Normal, expected inputs
Edge cases: Empty inputs, very long inputs, special characters
Adversarial: Inputs designed to break or confuse the model
Format compliance: Does output match the requested structure?

Automated Evaluation Methods

String Matching

Simple but limited — check if output contains expected elements:

python

def evaluate_output(output, expected_keywords):
    score = 0
    for keyword in expected_keywords:
        if keyword.lower() in output.lower():
            score += 1
    return score / len(expected_keywords)

LLM-as-Judge

Use another LLM call to evaluate the output:

You are an expert evaluator. Rate the following AI response
on a scale of 1-5 for each criterion:

Original question: {question}
AI response: {response}

Rate:
1. Accuracy (1-5):
2. Helpfulness (1-5):
3. Clarity (1-5):
4. Completeness (1-5):

Provide a brief justification for each rating.

This is one of the most effective automated evaluation methods available today.

Code Execution Tests

For code generation, actually run the generated code:

python

def evaluate_code_output(generated_code, test_inputs, expected_outputs):
    try:
        exec(generated_code)
        results = []
        for inp, expected in zip(test_inputs, expected_outputs):
            actual = eval(f"solution({inp})")
            results.append(actual == expected)
        return sum(results) / len(results)
    except Exception as e:
        return 0.0  # Code doesn't even run

The Iteration Process

Step 1: Establish Baseline

Run your initial prompt against all test cases. Record scores.

Prompt v1: "Write a Python function to [task]"

Test Case 1: 3/5
Test Case 2: 2/5
Test Case 3: 4/5
Test Case 4: 1/5
Average: 2.5/5

Step 2: Identify Failure Patterns

Look at where the prompt fails:

Missing edge case handling? → Add explicit instructions
Wrong format? → Add format examples
Incorrect logic? → Add chain-of-thought
Inconsistent? → Add constraints and examples

Step 3: Make One Change at a Time

Prompt v2: Added "Handle edge cases including empty input,
           single element, and duplicate values"

Test Case 1: 3/5  (no change)
Test Case 2: 4/5  (improved!)
Test Case 3: 4/5  (no change)
Test Case 4: 3/5  (improved!)
Average: 3.5/5    (+1.0 improvement)

Step 4: Repeat

Continue refining until you reach acceptable quality.

A/B Testing Prompts

Compare two prompt versions on the same test set:

Prompt A (Direct):
"Write a function that validates passwords."

Prompt B (Detailed):
"Write a Python function called validate_password that checks:
- Minimum 8 characters
- At least one uppercase, one lowercase, one digit
- At least one special character (!@#$%^&*)
Return a dict with 'valid': bool and 'errors': list[str]"

Metric	Prompt A	Prompt B
Accuracy	2.5/5	4.5/5
Format compliance	1/5	5/5
Edge case handling	2/5	4/5
Average	1.8	4.5

Prompt B wins clearly because of specificity.

Common Iteration Strategies

When output is too vague → Add specificity

Before: "Explain databases"
After:  "Compare PostgreSQL and MongoDB for an e-commerce app
         handling 10K daily orders. Cover: data modeling,
         query performance, scaling, and team expertise needed."

When output format is wrong → Add explicit format

Before: "List the pros and cons"
After:  "List pros and cons in this format:
         ✅ Pro: [description]
         ❌ Con: [description]
         Provide exactly 3 of each."

When reasoning is flawed → Add CoT

Before: "What's the time complexity?"
After:  "Analyze the time complexity step by step:
         1. Identify the loops and recursive calls
         2. Count operations for each
         3. Combine into Big O notation
         4. Explain in plain English"

When output is inconsistent → Add examples

Before: "Classify this ticket"
After:  "Classify this support ticket into exactly one category.

         Examples:
         'I can't log in' → Authentication
         'The page is slow' → Performance
         'Add dark mode' → Feature Request

         Ticket: [input]
         Category:"

Versioning Your Prompts

Track prompt versions like code:

python

PROMPTS = {
    "code_review": {
        "version": "2.3",
        "template": """...""",
        "changelog": [
            "v1.0: Initial prompt",
            "v1.1: Added security check requirement",
            "v2.0: Restructured with persona pattern",
            "v2.1: Added severity ratings",
            "v2.2: Fixed inconsistent format",
            "v2.3: Added edge case examples"
        ],
        "avg_score": 4.2,
        "test_cases": 15
    }
}

Evaluation Checklist

Before deploying a prompt to production:

Tested with at least 10 diverse inputs
Edge cases handled (empty, very long, special characters)
Output format is consistent across runs
Adversarial inputs don't break it
Performance is acceptable (latency, token usage)
Cost per request is within budget
Compared against at least one alternative prompt
Documented with version history

Key Takeaways

Measure, don't guess — Use scoring rubrics and test suites
Change one thing at a time — Isolate what helps
Test edge cases — Real-world inputs are messy
Version your prompts — Track what changed and why
Use LLM-as-judge — For scalable automated evaluation
Keep iterating — Good prompts evolve over time

With strong prompt engineering skills, you're ready to start building with AI APIs. The next module covers working with OpenAI, Anthropic, and Google's APIs.