ln-811-performance-profiler

Paths: File paths (references/, ../ln-*) are relative to this skill directory.

ln-811-performance-profiler

Type: L3 Worker Category: 8XX Optimization

Runtime profiler that executes the optimization target, measures multiple metrics (CPU, memory, I/O, time), instruments code for per-function breakdown, and produces a standardized performance map from real data.

Overview

| Aspect | Details | |--------|---------| | Input | Problem statement: target (file/endpoint/pipeline) + observed metric | | Output | Performance map (multi-metric, per-function), suspicion stack, bottleneck classification | | Pattern | Discover test → Baseline run → Static analysis → Deep profile → Performance map → Report |

Workflow

Phases: Test Discovery → Baseline Run → Static Analysis → Deep Profile → Performance Map → Report

Phase 0: Test Discovery/Creation

MANDATORY READ: Load references/ci_tool_detection.md for test framework detection. MANDATORY READ: Load references/benchmark_generation.md for auto-generating benchmarks when none exist.

Find or create commands that exercise the optimization target. Two outputs: test_command (profiling/measurement) and e2e_test_command (functional safety gate).

Step 1: Discover test_command

| Priority | Method | Action | |----------|--------|--------| | 1 | User-provided | User specifies test command or API endpoint | | 2 | Discover existing E2E test | Grep test files for target entry point (stop at first match) | | 3 | Create test script | Generate per references/benchmark_generation.md to .hex-skills/optimization/{slug}/profile_test.sh |

E2E discovery protocol (stop at first match):

| Priority | Method | How | |----------|--------|-----| | 1 | Route-based search | Grep e2e/integration test files for entry point route | | 2 | Function-based search | Grep for entry point function name | | 3 | Module-based search | Grep for import of entry point module |

Test creation (if no existing test found):

| Target Type | Generated Script | |-------------|-----------------| | API endpoint | curl -w "%{time_total}" -o /dev/null -s {endpoint} | | Function | Stack-specific benchmark per references/benchmark_generation.md | | Pipeline | Full pipeline invocation with test input |

Step 2: Discover e2e_test_command

If test_command came from E2E discovery (Step 1 priority 2): e2e_test_command = test_command.

Otherwise, run E2E discovery protocol again (same 3-priority table) to find a separate functional safety test.

If not found: e2e_test_command = null, log: WARNING: No e2e test covers {entry_point}. Full test suite serves as functional gate.

Output

| Field | Description | |-------|-------------| | test_command | Command for profiling/measurement | | e2e_test_command | Command for functional safety gate (may equal test_command, or null) | | e2e_test_source | Discovery method: user / route / function / module / none |

Phase 1: Baseline Run (Multi-Metric)

Run test_command with system-level profiling. Capture simultaneously:

| Metric | How to Capture | When | |--------|---------------|------| | Wall time | time wrapper or test harness | Always | | CPU time (user+sys) | /usr/bin/time -v or language profiler | Always | | Memory peak (RSS) | /usr/bin/time -v (Max RSS) or tracemalloc / process.memoryUsage() | Always | | I/O bytes | /usr/bin/time -v or structured logs | If I/O suspected | | HTTP round-trips | Count from structured logs or application metrics | If network I/O in call graph | | GPU utilization | nvidia-smi --query-gpu | Only if CUDA/GPU detected in stack |

Baseline Protocol

| Parameter | Value | |-----------|-------| | Runs | 3 | | Metric | Median | | Warm-up | 1 discarded run | | Output | baseline — multi-metric snapshot |

Monitor for Profiler Runs (Claude Code 2.1.98+)

MANDATORY READ: Load references/monitor_integration_pattern.md

During baseline and deep profile runs: Monitor(command="{test_command} 2>&1", timeout_ms=300000, description="profiler run N")

Detect crashes or infinite loops immediately. Fallback: Bash with timeout.

Phase 2: Static Analysis → Instrumentation Points

MANDATORY READ: Load [bottleneck_classification.md](references/bottleneck_classification.md)

Trace call chain from code + build suspicion stack. Purpose: guide WHERE to instrument in Phase 3.

Step 1: Trace Call Chain

Starting from entry point, trace depth-first (max depth 5). At each step, READ the full function body.

Cross-service tracing: If service_topology is available from coordinator and a step makes an HTTP/gRPC call to another service whose code is accessible:

| Situation | Action | |-----------|--------| | HTTP call to service with code in submodule/monorepo | Follow into that service's handler: resolve route → trace handler code (depth resets to 0 for the new service) | | HTTP call to service without accessible code | Classify as External, record latency estimate | | gRPC/message queue to known service | Same as HTTP — follow into handler if code accessible |

Record service: "{service_name}" on each step to track which service owns it. The performance_map steps tree can span multiple services.

Depth-First Rule: If code of the called service is accessible — ALWAYS profile INSIDE. NEVER classify an accessible service as "External/slow" without profiling its internals. "Slow" is a symptom, not a diagnosis.

5 Whys for each bottleneck: Before reporting a bottleneck, chain "why?" until you reach config/architecture level:

1. "What is slow?" → alignment service (5.9s) 2. "Why?" → 6 pairs × ~1s each 3. "Why ~1s per pair?" → O(n²) mwmf computation 4. "Why O(n²)?" → library default, not production config 5. "Why default?" → matching_methods not configured → root cause = config

Step 2: Classify & Suspicion Scan

For each step, classify by type (CPU, I/O-DB, I/O-Network, I/O-File, Architecture, External, Cache) and scan for performance concerns.

Suspicion checklist (minimum, not limitation):

| Category | What to Look For | |----------|-----------------| | Connection management | Client created per-request? Missing pooling? Missing reuse? | | Data flow | Data read multiple times? Over-fetching? Unnecessary transforms? | | Async patterns | Sync I/O in async context? Sequential awaits without data dependency? | | Resource lifecycle | Unclosed connections? Temp files? Memory accumulation in loop? | | Configuration | Hardcoded timeouts? Default pool sizes? Missing batch size config? | | Redundant work | Same validation at multiple layers? Same data loaded twice? | | Architecture | N+1 in loop? Batch API unused? Cache infra unused? Sequential-when-parallel? | | (open) | Anything else spotted — checklist does not limit findings |

Step 2b: Suspicion Deduplication

MANDATORY READ: Load references/output_normalization.md

After generating suspicions across all call chain steps, normalize and deduplicate per §1-§2:

• Normalize suspicion descriptions (replace specific values with placeholders)
• Group identical suspicions across different steps → merge into single entry with affected_steps: [list]
• Example: "Missing connection pooling" found in steps 1.1, 1.2, 1.3 → one suspicion with affected_steps: ["1.1", "1.2", "1.3"]

Step 3: Verify & Map to Instrumentation Points

FOR each suspicion:
  1. VERIFY: follow code to confirm or dismiss
  2. VERDICT: CONFIRMED → map to instrumentation point | DISMISSED → log reason
  3. For each CONFIRMED suspicion, identify:
     - function to wrap with timing
     - I/O call to count
     - memory allocation to track

Profiler Selection (per stack)

| Stack | Non-invasive profiler | Invasive (if non-invasive insufficient) | |-------|----------------------|----------------------------------------| | Python | py-spy, cProfile | time.perf_counter() decorators | |