Observability Designer

The agent designs production-ready observability strategies that combine the three pillars (metrics, logs, traces) with SLI/SLO frameworks, golden signals monitoring, and alert optimization.

Workflow

Catalogue services -- List every service in scope with its type (request-driven, pipeline, storage), criticality tier (T1-T3), and owning team. Validate that at least one T1 service exists before proceeding.
Define SLIs per service -- For each service, select SLIs from the Golden Signals table. Map each SLI to a concrete Prometheus/InfluxDB metric expression.
Set SLO targets -- Assign SLO targets based on criticality tier and user expectations. Calculate the corresponding error budget (e.g., 99.9% = 43.8 min/month).
Design burn-rate alerts -- Create multi-window burn-rate alert rules for each SLO. Validate that every alert has a clear runbook link and response action.
Build dashboards -- Generate dashboard specs following the hierarchy: Overview > Service > Component > Instance. Cap each screen at 7 panels. Include SLO target reference lines.
Configure log aggregation -- Define structured log format, set log levels, assign correlation IDs, and configure retention policies per tier.
Instrument traces -- Set up distributed tracing with sampling strategy (head-based for dev, tail-based for production). Define span boundaries at service and database call points.
Validate coverage -- Confirm every T1 service has metrics, logs, and traces. Confirm every alert has a runbook. Confirm dashboard load time is under 2 seconds.

SLI/SLO Quick Reference

| SLI Type | Metric Expression (Prometheus) | Typical SLO | |----------|-------------------------------|-------------| | Availability | 1 - (sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) | 99.9% | | Latency (P99) | histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) | < 500ms | | Error rate | sum(rate(grpc_server_handled_total{grpc_code!="OK"}[5m])) / sum(rate(grpc_server_handled_total[5m])) | < 0.1% | | Throughput | sum(rate(http_requests_total[5m])) | > baseline |

Error Budget Calculation

Error Budget = 1 - SLO target

Example (99.9% availability):
  Monthly budget = 30d x 24h x 60m x 0.001 = 43.2 minutes
  If 20 minutes consumed, remaining = 23.2 minutes (53.7% left)

Burn-Rate Alert Design

| Window | Burn Rate | Severity | Budget Consumed | |--------|-----------|----------|-----------------| | 5 min / 1 hr | 14.4x | Critical (page) | 2% in 1 hour | | 30 min / 6 hr | 6x | Warning (ticket) | 5% in 6 hours | | 2 hr / 3 day | 1x | Info (dashboard) | 10% in 3 days |

Rule: Every critical alert must have an actionable runbook. If no clear action exists, downgrade to warning.

Alert Classification

| Severity | Meaning | Response | Routing | |----------|---------|----------|---------| | Critical | Service down or SLO burn rate high | Page on-call immediately | PagerDuty escalation | | Warning | Approaching threshold, non-user-facing | Create ticket, fix in business hours | Slack channel | | Info | Deployment notification, capacity trend | Review in next standup | Dashboard only |

Alert Fatigue Prevention

Hysteresis: Set different thresholds for firing (e.g., > 90% CPU for 5 min) and resolving (e.g., < 80% CPU for 10 min).
Suppression: Suppress dependent alerts during known outages (e.g., suppress pod alerts when node is down).
Grouping: Group related alerts into a single notification (e.g., all pods in one deployment).
Precision over recall: A missed alert that would self-resolve is better than 50 false pages per week.

Golden Signals

| Signal | What to Monitor | Key Metrics | |--------|----------------|-------------| | Latency | Request duration | P50, P95, P99 response time; queue wait; DB query time | | Traffic | Request volume | RPS with burst detection; active sessions; bandwidth | | Errors | Failure rate | 4xx/5xx rates; error budget consumption; silent failures | | Saturation | Resource pressure | CPU/memory/disk utilization; queue depth; connection pool usage |

Dashboard Design Rules

Hierarchy: Overview (all services) > Service (one service) > Component (e.g., database) > Instance
Panel limit: Maximum 7 panels per screen to manage cognitive load
Reference lines: Always show SLO targets and capacity thresholds
Time defaults: 4 hours for incident investigation, 7 days for trend analysis
Role-based views: SRE (operational), Developer (debug), Executive (reliability summary)

Structured Log Format

{
  "timestamp": "2025-11-05T14:30:00Z",
  "level": "ERROR",
  "service": "payment-api",
  "trace_id": "abc123def456",
  "span_id": "789ghi",
  "message": "Payment processing failed",
  "error_code": "PAYMENT_TIMEOUT",
  "duration_ms": 5023,
  "customer_id": "cust_42",
  "environment": "production"
}

Log levels: DEBUG (local dev only), INFO (request lifecycle), WARN (degraded but functional), ERROR (failed operation), FATAL (service cannot continue).

Trace Sampling Strategies

| Strategy | When to Use | Trade-off | |----------|------------|-----------| | Head-based (10%) | Development, low-traffic services | Misses rare errors | | Tail-based | Production, high-traffic | Captures errors/slow requests; higher resource cost | | Adaptive | Variable traffic patterns | Adjusts rate based on load; more complex to configure |

Runbook Template

# Alert: [Alert Name]

## What It Means
[One sentence explaining the alert condition]

## Impact
[User-facing vs internal; affected services]

## Investigation Steps
1. Check dashboard: [link]  (1 min)
2. Review recent deploys: [link]  (2 min)
3. Check dependent services: [list]  (2 min)
4. Review logs: [query]  (3 min)

## Resolution Actions
- If [condition A]: [action]
- If [condition B]: [action]
- If unclear: Escalate to [team] via [channel]

## Post-Incident
- [ ] Update incident timeline
- [ ] File post-mortem if > 5 min user impact

Example: E-Commerce Payment Service Observability

service: payment-api
tier: T1 (revenue-critical)
owner: payments-team

slis:
  availability:
    metric: "1 - rate(http_5xx) / rate(http_total)"
    slo: 99.95%
    error_budget: 21.6 min/month
  latency_p99:
    metric: "histogram_quantile(0.99, http_duration_seconds)"
    slo: < 800ms
  error_rate:
    metric: "rate(payment_failures) / rate(payment_attempts)"
    slo: < 0.5%

alerts:
  - name: PaymentHighErrorRate
    expr: "rate(payment_failures[5m]) / rate(payment_attempts[5m]) > 0.01"
    for: 2m
    severity: critical
    runbook: "https://wiki.internal/runbooks/payment-errors"

dashboard_panels:
  - Payment success rate (gauge)
  - Transaction volume (time series)
  - P50/P95/P99 latency (time series)
  - Error breakdown by type (stacked bar)
  - Downstream dependency health (status map)
  - Error budget remaining (gauge)

Cost Optimization

Metric retention: 15-day full resolution, 90-day downsampled, 1-year aggregated
Log sampling: Sample DEBUG/INFO at 10% in high-throughput services; always keep ERROR/FATAL at 100%
Trace sampling: Tail-based sampling retains only errors and slow requests (> P99)
Cardinality management: Alert on any metric with > 10K unique label combinations

Scripts

SLO Designer (`slo_designer.py`)

Generates SLI/SLO frameworks from service description JSON. Outputs SLI definitions, SLO targets, error budgets, burn-rate alerts, and SLA recommendations.

Alert Optimizer (`alert_optimizer.py`)

Analyzes existing alert configurations for noise, coverage gaps, and duplicate rules. Outputs an optimization report with improved thresholds.

Dashboard Generator (`dashboard_generator.py`)

Creates Grafana-compatible dashboard JSON from service/system descriptions. Covers golden signals, RED/USE methods, and role-based views.

observability-designer

Como adicionar

Cole no README do seu repo

Skills relacionadas

internal-comms

smart-explore

babysit

do

Receba novas skills de DevOps e Infra toda segunda