When this skill is activated, always start your first response with the 🧢 emoji.

Backend Engineering

A senior backend engineer's decision-making framework for building production systems. This skill covers the six pillars of backend engineering - schema design, scalable systems, observability, performance, security, and API design - with an emphasis on when to use each pattern, not just how. Designed for mid-level engineers (3-5 years) who know the basics and need opinionated guidance on trade-offs.

When to use this skill

Trigger this skill when the user:

• Designs a database schema or plans a migration
• Chooses between monolith vs microservices or evaluates scaling strategies
• Sets up logging, metrics, tracing, or alerting
• Diagnoses a performance issue (slow queries, high latency, memory pressure)
• Implements authentication, authorization, or secrets management
• Designs a REST, GraphQL, or gRPC API
• Needs retry, circuit breaker, or idempotency patterns
• Plans data consistency across services (sagas, outbox, eventual consistency)

Do NOT trigger this skill for:

• Frontend-only concerns (CSS, React components, browser APIs)
• DevOps/infra provisioning (use a Terraform/Docker/K8s skill instead)

Key principles

1. Design for failure, not just success - Every network call can fail. Every disk can fill. Every dependency can go down. The question is not "will it fail" but "how does it degrade?" Design graceful degradation paths before writing the happy path.

2. Observe before you optimize - Never guess where the bottleneck is. Instrument first, measure second, optimize third. A 10ms query called 1000 times matters more than a 500ms query called once.

3. Simple until proven otherwise - Start with a monolith, a single database, and synchronous calls. Add complexity (microservices, queues, caches) only when you have evidence the simple approach fails. Every architectural boundary is a new failure mode.

4. Secure by default, not by afterthought - Auth, input validation, and encryption are not features to add later. They are constraints to build within from day one. Use established libraries. Never roll your own crypto.

5. APIs are contracts, not implementation details - Once published, an API is a promise. Design from the consumer's perspective inward. Version explicitly. Break nothing silently.

Core concepts

Backend engineering is the discipline of building reliable, performant, and secure server-side systems. The six pillars form a hierarchy:

Schema design is the foundation - get the data model wrong and everything built on top inherits that debt. Scalable systems define how components communicate and grow. Observability gives you eyes into what's actually happening in production. Performance is the art of making it fast after you've made it correct. Security is the set of constraints that keep the system trustworthy. API design is the surface area through which consumers interact with all of the above.

These pillars are not independent. A bad schema creates performance problems. Poor observability makes security incidents invisible. A poorly designed API forces clients into patterns that break your scaling strategy. Think of them as a connected system, not a checklist.

Common tasks

Design a database schema

Start from access patterns, not entity relationships. Ask: "What queries will this serve?" before drawing a single table.

Decision framework:

• Read-heavy, predictable queries -> Normalize (3NF), add targeted indexes
• Write-heavy, high throughput -> Consider denormalization, append-only tables
• Complex relationships with traversals -> Consider a graph model
• Unstructured/evolving data -> Document store (but think twice)

Indexing rule of thumb: Index columns that appear in WHERE, JOIN, and ORDER BY. A composite index on (a, b, c) serves queries on (a), (a, b), and (a, b, c) but NOT (b, c). Check the references/ file for detailed indexing strategies.

Always plan migration rollbacks. A deploy that adds a column is safe. A deploy that drops a column is a one-way door. Use expand-contract migrations for breaking changes.

Choose a scaling strategy

Is a single server sufficient?
  YES -> Stay there. Optimize vertically first.
  NO  -> Is the bottleneck compute or data?
    COMPUTE -> Horizontal scale with stateless services + load balancer
    DATA    -> Is it read-heavy or write-heavy?
      READ  -> Add read replicas, then caching layer
      WRITE -> Partition/shard the database

Only introduce microservices when you have: (a) independent deployment needs, (b) different scaling profiles per component, or (c) team boundaries that demand it.

Never split a monolith along technical layers (API service, data service). Split along business domains (orders, payments, inventory).

Set up observability

Implement the three pillars with correlation:

Define SLOs before writing alerts. An SLO like "99.9% of requests complete in <200ms" gives you an error budget. Alert when the burn rate threatens the budget, not on every spike.

Diagnose a performance issue

Follow this checklist in order:

1. Check metrics - is it CPU, memory, I/O, or network?
2. Check slow query logs - are there N+1 patterns or full table scans?
3. Check connection pools - are connections exhausted or leaking?
4. Check external dependencies - is a downstream service slow?
5. Profile the code - only after ruling out infrastructure causes

The fix for "the database is slow" is almost never "add more database." It's usually: add an index, fix an N+1, or cache a hot read path.

Secure a service

Minimum security checklist for any backend service:

• Authentication: Use OAuth 2.0 / OIDC for user-facing, API keys + HMAC for service-to-service. Never store plain-text passwords (bcrypt/argon2 minimum).
• Authorization: Implement at the middleware level. Default deny. Check permissions on every request, not just at the edge.
• Input validation: Validate at system boundaries. Use allowlists, not blocklists. Parameterize all SQL queries.
• Secrets: Use a secrets manager (Vault, AWS Secrets Manager). Never commit secrets to git. Rotate regularly.
• Transport: TLS everywhere. No exceptions.

Design an API

REST decision table:

Pagination: Use cursor-based for large/changing datasets, offset-based only for small/static datasets. Always include a next_cursor field.

Versioning: URL path versioning (/v1/) for public APIs, header versioning for internal. Never break existing consumers silently.

Rate limiting: Token bucket for user-facing, fixed window for internal. Always return Retry-After headers with 429 responses.

Handle partial failures

When services depend on other services, failures cascade. Use these patterns:

• Retry with exponential backoff + jitter - for transient failures (network blips, 503s). Cap at 3-5 retries.
• Circuit breaker - stop calling a failing dependency. States: closed (normal) -> open (failing, fast-fail) -> half-open (testing recovery).
• Idempotency keys - make retries safe. Every mutating operation should accept an idempotency key so duplicate requests produce the same result.
• Timeouts - always set them.