devops

DevOps Engineer

CI/CD, containers, Kubernetes, IaC, and observability patterns for production systems.

Table of Contents

• [GitHub Actions](#github-actions)
• [Docker](#docker)
• [Kubernetes](#kubernetes)
• [Terraform / OpenTofu](#terraform--opentofu)
• [Observability (OpenTelemetry)](#observability-opentelemetry)
• [GitOps](#gitops)
• [Security Scanning](#security-scanning)
• [Cost Optimization](#cost-optimization)

GitHub Actions

Secretless Auth via OIDC (no long-lived credentials)

# .github/workflows/deploy.yml
permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production   # requires manual approval in repo settings
    steps:
      - uses: actions/checkout@v4

      # AWS
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: us-east-1

      # GCP
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: projects/123/locations/global/workloadIdentityPools/github/providers/github
          service_account: deploy@my-project.iam.gserviceaccount.com

Pin actions to full commit SHA (not tags)

# WRONG — tag can be moved
- uses: actions/checkout@v4

# RIGHT — immutable, auditable
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2

Reusable workflows

# .github/workflows/_build.yml (reusable)
on:
  workflow_call:
    inputs:
      image-tag:
        required: true
        type: string
    secrets:
      REGISTRY_TOKEN:
        required: true

# Caller
jobs:
  build:
    uses: ./.github/workflows/_build.yml
    with:
      image-tag: ${{ github.sha }}
    secrets:
      REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}

Cache dependencies

- uses: actions/setup-node@v4
  with:
    node-version: '22'
    cache: 'npm'

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

Docker

Multi-stage build with distroless

# Build stage
FROM node:22-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build

# Production stage — distroless has no shell, minimal attack surface
FROM gcr.io/distroless/nodejs22-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["dist/server.js"]

BuildKit caching and secrets (never bake secrets into layers)

# syntax=docker/dockerfile:1

# Mount pip cache — not stored in image
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# Mount secret at build time — never appears in layer history
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

# Build with BuildKit
DOCKER_BUILDKIT=1 docker build \
  --secret id=npmrc,src=.npmrc \
  --cache-from type=registry,ref=myrepo/app:cache \
  --cache-to   type=registry,ref=myrepo/app:cache,mode=max \
  -t myrepo/app:$GIT_SHA .

Layer ordering (slow → fast)

# 1. OS deps (rarely change)
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

# 2. Package manager files (change on dep updates)
COPY package*.json ./
RUN npm ci --ignore-scripts

# 3. Source code (changes every commit)
COPY . .
RUN npm run build

Kubernetes

Resource requests/limits (always set both — determines QoS class)

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    # Never set CPU limits — causes throttling under burst load
    # Set only memory limits to get Burstable QoS
    memory: "512Mi"

QoS classes:

• Guaranteed: requests == limits for all containers → last evicted
• Burstable: requests < limits → evicted after BestEffort
• BestEffort: no requests/limits → first evicted

All three probe types (on separate endpoints)

livenessProbe:
  httpGet:
    path: /healthz/live      # returns 200 if process is alive; restart if fails
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /healthz/ready     # returns 200 if ready to receive traffic; remove from LB if fails
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /healthz/startup   # gives slow-start apps time before liveness kicks in
    port: 8080
  failureThreshold: 30       # 30 × 10s = 5 minutes max startup
  periodSeconds: 10

Rule: Never reuse the same endpoint for all three probes. Liveness must not depend on downstream services — it will cause cascading restarts.

HPA + KEDA for event-driven scaling

# Standard HPA (CPU/memory)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min before scaling down

# KEDA — scale on queue depth, Kafka lag, cron, etc.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0    # can scale to zero
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123/my-queue
        queueLength: "10"   # 1 pod per 10 messages

PodDisruptionBudget (required for zero-downtime deploys)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2        # or use maxUnavailable: 1
  selector:
    matchLabels:
      app: api

Network policies (default deny, explicit allow)

# Default deny all ingress/egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

# Allow only: api → postgres on 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-postgres
spec:
  podSelector:
    matchLabels:
      app: postgres
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - port: 5432

Spread across zones

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api

Terraform / OpenTofu

Layered module structure

infra/
├── modules/              # reusable building blocks (no state)
│   ├── vpc/
│   ├── eks/
│   └── rds/
├── environments/
│   ├── staging/          # thin wrappers that call modules
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── backend.tf    # remote state config
│   └── production/
└── shared/               # cross-env resources (DNS, ECR)

Remote state + locking

# backend.tf
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "production/eks/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"   # prevents concurrent applies
    encrypt        = true
  }
}

Pin providers and modules

terraform {
  required_version = ">= 1.9, < 2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"   # allows 5.x but not 6.x
    }
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.13.0"   # exact pin for production
}

Drift detection

# Check for infrastructure drift without changing anything
terraform plan -refresh-only

# In CI — fail pipeline if d