observability

Overview

Observability is the ability to understand the internal state of a system from its external outputs. For AI systems this is especially critical: agents make decisions that are hard to interpret without detailed telemetry.

The three pillars: Logs (what happened), Traces (how long and where), Metrics (aggregate health).

When to Use

• Before deploying any new service to production
• When adding AI agent capabilities to an existing system
• When debugging production issues
• When designing multi-agent pipelines

Process

Step 1: Structured Logging

1. All logs must be structured (JSON, not free text). Fields: timestamp, level, service, traceId, message, context.
2. Log levels used correctly:
   • ERROR: Something failed that requires immediate attention
   • WARN: Something unexpected happened but the system recovered
   • INFO: Normal significant events (requests received, jobs completed)
   • DEBUG: Detailed diagnostic information (off in production by default)
3. Never log secrets, PII, or auth tokens.
4. For AI systems, log: prompt inputs (sanitized), model outputs, token counts, latency, model version.

Verify: Logs are structured JSON. No secrets in logs. AI interactions logged.

Step 2: Distributed Tracing

5. Every request gets a unique traceId generated at the entry point.
6. traceId is propagated through all downstream calls (HTTP headers, message queues, agent calls).
7. Each service/agent creates a span for its work, with: start time, end time, parent span ID.
8. Use OpenTelemetry as the standard instrumentation library.

Verify: You can trace a single request across all services/agents in a single view.

Step 3: Metrics

9. Define and track key metrics:
   • RED metrics: Rate (requests/sec), Errors (error rate %), Duration (latency p50/p95/p99)
   • AI-specific: Token usage, prompt cost, model latency, hallucination rate, retrieval precision
10. Dashboards: one dashboard per service with RED metrics, one dashboard for AI system health.

Verify: RED metrics are tracked for every service. AI-specific metrics tracked for AI systems.

Step 4: Alerting

11. Alerts must be actionable — every alert should have a runbook.
12. Alert on symptoms (high error rate, high latency), not just causes.
13. AI-specific alerts: token budget exceeded, model error rate spike, retrieval failure rate spike.
14. On-call rotation: someone is responsible for every alert at all times.

Verify: Every alert has a runbook. On-call rotation defined.

Common Rationalizations (and Rebuttals)

Verification

• Structured JSON logging on all services
• No secrets in logs
• Distributed tracing with trace ID propagation
• RED metrics tracked for all services
• AI-specific metrics tracked (tokens, cost, latency)
• Alerts configured with runbooks

References

• production-deployment skill
• multi-agent-orchestration skill
• OpenTelemetry documentation