Tight feedback loops are everything in this work, but you need to put in the groundwork to get there.

Without a clear problem definition and fit, and real data, you'll get stuck fast.

This article is about building both: the foundation that makes iteration meaningful, and the environment that allows you to close the loop.

• Inputs are messy: free text, images, or data with an indeterminate shape (user-submitted descriptions, search queries in broken English, photos of unknown items)
• The transformation requires judgment that's hard to encode in plain-code rules
• A "Pretty good" solution is valuable, and mistakes can be caught, corrected or tolerated
• Simpler approaches have already hit their limits
• Latency and cost rule out multi-step chains or long reasoning

• The transformation is describable as a clear algorithm
• The solution exists as another AI-centric approach, for example leveraging embeddings, a small pre-trained classifier or an off-the-shelf model
• Wrong outputs are costly and hard to detect
• You're reaching for an LLM because it's exciting, not because you need it
• You don't yet have a clear picture of your input space or what the output should look like
• You haven't mapped out the breadth of inputs or defined the shape of the output

Arguably the most important step, since it informs everything else.

Gather a wide range of expected inputs based on anticipated user behavior. Real data, not toy examples. Be extremely careful not to cherry-pick "good" examples. You need the messy ones, the weird ones, the adversarial ones.

For our HN analyzer: pull actual comment histories. The prolific Rust expert. The lurker with three generic replies. The user whose tone shifts between helpful and combative. The troll. The brand new account.

Include abuse cases and scenarios where you want to gate early and avoid calling the LLM at all. Accounts with too few comments, Comments that are all just "+1". The over-promoter.

From there, discuss and manually generate expected outputs for your initial input set. Then take some hammock time.

This is where you will begin to nail down the structure: what each part of the output will be named, and descriptions for each of those values.

For example, an early output shape for our use case might be

{
  "summary": "Backend engineer focused on distributed systems...",
  "expertise_areas": ["rust", "databases"],
  "topics": [{"name": "rust", "count": 45}, ...],
  "sentiment_over_time": [{"month": "2024-01", "avg_sentiment": 0.7}, ...],
  "overall_tone": "helpful",
  "confidence": "high"
}

Now write it out for your edge cases. What's the output for the troll? The lurker? The user who shifts tone?

Doing this work now saves you from discovering these gaps mid-iteration. These manually-created pairs become your test cases, your few-shot examples, your evaluation sets.

This work is never wasted.

A quick but important side discussion.

Initially, I treated prompts as metadata. Files to be loaded in at runtime, and kept aside from the codebase. I've come around in my thinking though that for these kinds of systems, prompts should be treated as code.

They should be versioned along with the codebase. They should be reviewed along with evidence and reasoning as to why the wording changed. They should be tested using code-like machinery.

They should NOT be managed and developed using third-party tooling. Using any useful tool to get the job done is fine, but the main source-of-truth should live in your codebase.

This means:

• The same prompt definitions used in your exploration tools are the ones used in production
• Changes to prompts go through the same review process as other code changes
• Prompts live in your repository, not in a dashboard, not in a vendor platform, not scattered across notebooks

This approach pays off in consistency and traceability. When something changes in production behavior, you can trace it back to a specific commit.

The payoff is consistency and traceability. When production behavior changes, you can trace it back to a specific commit. When you're debugging a weird output, you know exactly what prompt produced it, not "whatever was in that config file at the time.

If you need variations on prompts, treat it like an alternative branch in your codebase.

Get a manually deployable minimal POC in place. In the era of vibe coding, this becomes fun and fast. Speed matters more than polish here.

Important points are:

• Use the same prompt code that will be used in production
• Allow everyone involved to tweak, explore, and discuss
• Produce shareable results as links
  • Include the prompt used as data on the page
  • Include the inputs
  • Include raw outputs
  • Include raw execution metadata like tokens used and latency

This becomes your central tool for collaborative refinement. When someone finds an interesting edge case or a particularly good output, they can share a link that captures exactly how it was produced.

from pydantic import BaseModel, Field

class Topic(BaseModel):
    name: str = Field(description="Lowercase, hyphenated topic name")
    count: int = Field(description="Number of comments touching this topic")

class Quote(BaseModel):
    text: str = Field(description="A short, punchy excerpt (max 150 chars)")
    context: str = Field(description="Brief context about what they were discussing")

class HNProfileAnalysis(BaseModel):
    headline: str = Field(
        description="A punchy one-line summary of who this person is on HN "
                    "(e.g. 'Security expert who loves to debate')"
    )
    key_observations: list[str] = Field(
        description="3-5 bullet point observations about this user's HN presence"
    )
    expertise_areas: list[str] = Field(
        description="Up to 5 areas where the user demonstrates clear knowledge"
    )
    topics: list[Topic] = Field(
        description="Top 5-8 topics by comment frequency"
    )
    notable_quotes: list[Quote] = Field(
        description="2-3 memorable or representative quotes from their comments"
    )
    overall_tone: str = Field(
        description="One of: helpful, neutral, combative, mixed, analytical, enthusiastic"
    )
    engagement_style: str = Field(
        description="How they typically engage: e.g. 'Long detailed explanations', "
                    "'Quick corrections', 'Asks probing questions'"
    )
    confidence: str = Field(
        description="high if 20+ comments, medium if 5-20, low if <5"
    )

Create a layer of simple, fast-running tests that check for obviously bad behavior:

• Output length exceeding acceptable limits
• Response latency beyond thresholds
• Presence of forbidden words or patterns (regex-based)
• Malformed JSON or schema violations

These tests should run quickly and catch the most egregious failures before anything else. They are your first line of defense.

After a round of interactive refinement, capture your identified good and bad input/output pairs in YAML files. These become pytest-driven integration tests that:

• Run against the actual prompt code
• Verify that known-good inputs still produce acceptable outputs
• Confirm that known-bad inputs are handled appropriately
• Can be extended as new edge cases are discovered

This creates a growing suite of regression tests that protect against prompt changes breaking previously working cases.

Generate static HTML output of bulk runs that can be shared with stakeholders. Single, self-contained HTML pages with small JavaScript snippets for filtering or sorting by category are excellent for gathering feedback.

These don't need to be fancy. A table of inputs and outputs, maybe color-coded by confidence or tone, is enough for a PM or reviewer to scan through and flag issues.

Try to make your POC hosted and sharable. Results should be shareable as links. These links should capture:

• The exact inputs used
• How they were templated
• The system prompt and schema at the time
• Model settings (model, temperature)
• Raw outputs alongside the "presented" output

This is vital for distributing the work of finding edge cases. When a colleague stumbles on a weird output, they can send you a link instead of a screenshot. You see exactly what they saw, with full context for debugging.

It also distributes the joy of exploring the solution.

Getting others hands-on early helps ensure you're solving a "right problem", not just a problem you assumed you had.

Turn your input examples into plain text fixtures in a predictable format. For example, plain YAML or JSON files, checked into your repo.

# fixtures/inputs/prolific_rust_expert.yaml
username: "rustfan42"
comments:
  - text: "The borrow checker isn't the enemy..."
    story: "Why I stopped mass-promoting Rust"
    time: "2024-01-15"
  # ...

# fixtures/inputs/troll_account.yaml
username: "angryposter"
comments:
  - text: "This is stupid and you're stupid"
    story: "Show HN: My weekend project"
    time: "2024-02-01"
  # ...

The outputs you manually wrote during data gathering become your expected results.

They can be used to drive fast, static tests (like should_gate or comparing enum values), or confirm using simple LLM-as-a-judge expected results.

# fixtures/expected/prolific_rust_expert.yaml
headline_contains: ["rust", "expert"]
tone: "helpful"
confidence: "high"
topics_include: ["rust", "systems"]

# fixtures/expected/troll_account.yaml
stop_word: "stupid"
should_gate: true  # don't even call the LLM

Start simple. Many checks don't need an LLM at all:

@pytest.mark.parametrize("case", load_fixtures("fixtures/"))
def test_analysis(case):
    input_data = case["input"]
    expected = case["expected"]
    
    # Gate check - should we even call the LLM?
    if expected.get("should_gate"):
        assert should_gate(input_data), "Should have gated early"
        return
    
    # Run the analysis
    result = analyze_profile(input_data)
    
    # Static checks
    if "stop_words" in expected:
        for word in expected["stop_words"]:
            assert word not in result["headline"].lower()
    
    if "tone" in expected:
        assert result["overall_tone"] == expected["tone"]
    
    if "confidence" in expected:
        assert result["confidence"] == expected["confidence"]
    
    if "topics_include" in expected:
        for topic in expected["topics_include"]:
            assert topic in result["topics"]

Fast, deterministic, no API calls. These catch the obvious regressions, and should be easy enough to run as part of your iteration tooling, or locally as you get closer to a new solution.

Fast is key. Fast is fun.

For fuzzier checks, add an LLM-as-judge layer.

When I set this up, all judges share the same output schema and a single user prompt template which takes the inputs and outputs generated. I use a heavier model. I output a boolean and, if it fails, a reason and relevant snippet.

class JudgeReason(BaseModel):
    reason: str = Field(description="Why this passed or failed")
    snippet: str = Field(description="Relevant snippet from the output")


class JudgeVerdict(BaseModel):
    passes: bool = Field(description="Does the output meet the criteria?")
    reasons: list[JudgeReason] = Field(description="Supporting reasons")

USER_PROMPT = "## Input\n{input}\n\n## LLM Output\n{output}"

JUDGES = {
    "mentions_expertise": {
        "system_prompt": "You evaluate whether an LLM-generated profile headline mentions the user's area of expertise based on their comments.",
    },
    "tone_matches_comments": {
        "system_prompt": "You evaluate whether the assessed tone matches the actual tone of the source comments.",
    },
    "topics_relevant": {
        "system_prompt": "You evaluate whether the extracted topics are actually present in the source comments.",
    },
}

# fixtures/prolific_rust_expert.yaml
input:
  username: "rustfan42"
  comments:
    - text: "The borrow checker isn't the enemy..."
      story: "Why I stopped mass-promoting Rust"
judges:
  - mentions_expertise
  - tone_matches_comments
  - topics_relevant

def run_judge(judge_name: str, input_data: dict, llm_output: dict) -> JudgeVerdict:
    judge = JUDGES[judge_name]
    return call_llm(
        system_prompt=judge["system_prompt"],
        user_prompt=USER_PROMPT.format(
            input=json.dumps(input_data, indent=2),
            output=json.dumps(llm_output, indent=2)
        ),
        schema=JudgeVerdict,
        model=judge.get("model", "gpt-5.2")
    )

def load_fixtures():
    return [
        yaml.safe_load(f.read_text())
        for f in Path("fixtures").glob("*.yaml")
    ]

def fixture_judge_pairs():
    return [
        pytest.param(fixture, judge_name, id=f"{fixture['name']}/{judge_name}")
        for fixture in load_fixtures()
        for judge_name in fixture.get("judges", [])
    ]

@pytest.mark.parametrize("fixture,judge_name", fixture_judge_pairs())
def test_profile_judge(fixture, judge_name):
    input_data = fixture["input"]
    output = generate_profile(input_data)

    verdict = run_judge(judge_name, input_data, output)

    failures = [
        f"{r.reason} | snippet: {r.snippet!r}"
        for r in verdict.reasons
        if not verdict.passes
    ]
    assert verdict.passes, "\n".join(failures)

PASSED test_profile_judge[prolific_rust_expert/tone_matches_comments]
PASSED test_profile_judge[prolific_rust_expert/topics_relevant]
FAILED test_profile_judge[prolific_rust_expert/mentions_expertise]

Third-party eval platforms have their place, but they come with integration overhead, vendor lock-in, and workflows that may not match yours.

In the era of vibe coding and agent-assisted development, standing up your own test harness is fast. Plain text fixtures, pytest and a single web-app or multiple which support the team's exploration of the problem space.

These should be strongly considered, but loosely held. They should live in the same codebase, or the codebase should be designed such that the tooling can re-use the prompt definitions from the production code.

Aim to make the tooling easy to understand, but have depth for when you need to dig in and explore.

Sharability of results pays off in dividends always.

As your feeling for the problem space grows, your best fixtures can double as few-shot examples in the prompt itself:

def load_few_shot_examples(fixture_dir: str, names: list[str]) -> str:
    examples = []
    for name in names:
        case = load_yaml(f"{fixture_dir}/{name}.yaml")
        if case["include_as_few_shot"]:
            examples.append(
                f"Input:\n{format_input(case['input'])}\n\n"
                f"Output:\n{json.dumps(case['expected'], indent=2)}"
            )
    return "\n\n---\n\n".join(examples)

Same data, multiple uses. Your manually-written expected outputs become the examples that guide the model.

Your fixtures become documentation too.

When another engineer picks up this feature, they look at the fixture files and immediately understand what kinds of inputs the system handles, what the outputs look like, what edge cases exist.

No separate docs to maintain. The fixtures are the spec.