ai-observability-promptfoo

Promptfoo Patterns

Quick Guide: Use promptfoo for systematic LLM evaluation. Define prompts, providers, and test cases in promptfooconfig.yaml. Use assertion types (contains, is-json, llm-rubric, similar, cost, latency) to validate outputs. Use promptfoo eval to run (exits with code 100 on test failures), promptfoo view for results UI. Use model-graded assertions (llm-rubric, factuality) for subjective quality. Use promptfoo redteam run for security scanning. Use --share flag or promptfoo share to share results. All provider API keys come from environment variables -- never hardcode them.

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST define test cases with explicit assert arrays -- tests without assertions only capture output without validating it)

(You MUST use llm-rubric for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)

(You MUST set threshold on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)

(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)

(You MUST verify promptfoo eval exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)

</critical_requirements>

Auto-detection: promptfoo, promptfooconfig, promptfooconfig.yaml, promptfoo eval, promptfoo view, promptfoo redteam, llm-rubric, model-graded-closedqa, promptfoo share, promptfoo cache, assertion type, LLM evaluation, prompt testing, red teaming, PROMPTFOO_CONFIG

When to use:

• Writing or evaluating LLM prompts across one or more providers
• Setting up automated test suites for LLM-powered features
• Comparing model outputs side-by-side (GPT vs Claude vs Gemini)
• Running model-graded evaluations (LLM-as-a-judge)
• Red teaming LLM applications for security vulnerabilities
• Integrating LLM quality gates into CI/CD pipelines
• Validating structured output (JSON, function calls) from LLMs

Key patterns covered:

• promptfooconfig.yaml structure (prompts, providers, tests, defaultTest)
• Assertion types (deterministic, model-graded, performance)
• Custom TypeScript providers
• Red teaming configuration (plugins, strategies)
• CI/CD integration with GitHub Actions
• Programmatic API (evaluate() function)
• Result sharing and caching

When NOT to use:

• Unit testing application code (use your test runner)
• Load testing / benchmarking API throughput (use a load testing tool)
• Runtime monitoring of production LLM calls (use observability tooling)

Examples Index

• [Core: Config & Assertions](examples/core.md) -- promptfooconfig.yaml structure, providers, prompts, test cases, assertion types
• [Model-Graded & Advanced Assertions](examples/model-graded.md) -- llm-rubric, factuality, similar, context evaluation, custom assertions
• [Red Teaming](examples/red-teaming.md) -- Security scanning, plugins, strategies, presets
• [Custom Providers & Programmatic API](examples/custom-providers.md) -- TypeScript providers, evaluate() function, CI/CD integration
• [Quick API Reference](reference.md) -- CLI commands, assertion type table, provider IDs, red team plugins

<philosophy>

Philosophy

Promptfoo brings test-driven development to LLM applications. Instead of manually checking outputs, you define expected behaviors as assertions and run them systematically across prompts and providers.

Core principles:

1. Declarative test definitions -- YAML config over imperative test scripts. Define prompts, providers, test cases, and assertions in promptfooconfig.yaml. No code required for standard evaluations.
2. Assertion-driven validation -- Every test case should have assertions. Deterministic assertions (contains, is-json, equals) for structured output; model-graded assertions (llm-rubric, factuality) for subjective quality.
3. Comparative evaluation -- Run the same tests across multiple providers or prompt variants simultaneously. The results matrix shows which combination performs best.
4. Shift-left LLM testing -- Catch prompt regressions in CI before they reach production. promptfoo eval exits with code 100 on test failures, making it a natural CI quality gate.
5. Red teaming as a first-class concern -- Security scanning for prompt injection, PII leakage, harmful content, and jailbreak vulnerabilities is built in, not bolted on.

</philosophy>

<patterns>

Core Patterns

Pattern 1: Basic Configuration

Every promptfoo project starts with promptfooconfig.yaml. Three required sections: prompts, providers, tests.

# promptfooconfig.yaml
description: "Translation quality evaluation"

prompts:
  - "Convert the following to {{language}}: {{input}}"

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-sonnet-4-6

tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: icontains
        value: "bonjour"
      - type: llm-rubric
        value: "Output is a natural French translation, not word-for-word"

Why good: Declarative config, multi-provider comparison, both deterministic and model-graded assertions

# BAD: Tests without assertions
tests:
  - vars:
      language: French
      input: Hello world
  # No assert array -- output is captured but never validated

Why bad: Tests without assertions only log output, they never fail -- you lose the entire point of automated evaluation

See: [examples/core.md](examples/core.md) for prompts from files, provider config, defaultTest, variable loading from CSV

Pattern 2: Deterministic Assertions

Use for outputs with predictable, verifiable structure.

assert:
  # String matching
  - type: contains
    value: "error"
  - type: icontains # case-insensitive
    value: "success"
  - type: not-contains
    value: "internal server error"
  - type: starts-with
    value: "{"
  - type: regex
    value: "\\d{4}-\\d{2}-\\d{2}" # date pattern

  # Structured output
  - type: is-json
  - type: contains-json
  - type: is-valid-openai-tools-call

  # Performance
  - type: cost
    threshold: 0.01 # max $0.01 per call
  - type: latency
    threshold: 5000 # max 5 seconds

Why good: Fast, deterministic, no LLM cost for evaluation, catches structural regressions immediately

# BAD: Using llm-rubric for JSON validation
assert:
  - type: llm-rubric
    value: "Output must be valid JSON"

Why bad: Expensive (requires LLM call), slower, non-deterministic -- is-json does this deterministically for free

See: [examples/core.md](examples/core.md) for all deterministic assertion types with examples

Pattern 3: Model-Graded Assertions

Use for subjective quality where deterministic checks cannot capture intent.

assert:
  - type: llm-rubric
    value: "Response is helpful, accurate, and conversational in tone"
    provider: openai:gpt-4o

  - type: factuality
    value: "The capital of France is Paris. It has a population of ~2.1 million."
    provider: openai:gpt-4o

  - type: similar
    value: "The weather in Paris is sunny today"
    threshold: 0.8

  - type: model-graded-closedqa
    value: "Paris is the capital of France"
    provider: openai:gpt-4o

Why good: Evaluates subjective quality that deterministic assertions cannot capture, configurable grading provider

# BAD: No threshold on similar assertion
assert:
  - type: similar
    value: "expected output"
    # Missing threshold -- uses default which may be too lenient or strict

Why bad: Default similarity threshold may not match your quality bar, always set it explicitly

See: [examples/model-g