GitOps Patterns
Best practices for operating infrastructure and applications through Git as the single source of truth. Covers ArgoCD, Flux, Argo Rollouts, environment promotion, secret management, and progressive delivery.
GitOps Principles
GitOps extends Infrastructure as Code by adding a reconciliation loop that continuously ensures the live system matches the declared state in Git.
| Principle | Description | Implementation | |-----------|-------------|----------------| | Declarative | Entire system described declaratively | Kubernetes manifests, Helm charts, Kustomize overlays | | Versioned and immutable | Desired state stored in Git | Git commits as audit trail, tags for releases | | Pulled automatically | Software agents pull desired state | ArgoCD, Flux controllers watch Git repos | | Continuously reconciled | Agents correct drift automatically | Reconciliation loop detects and fixes divergence |
| GitOps vs Traditional CI/CD | GitOps | Traditional CI/CD | |-----------------------------|--------|-------------------| | Deployment trigger | Git commit (pull-based) | Pipeline step (push-based) | | Source of truth | Git repository | Pipeline state / manual | | Drift handling | Auto-corrected | Undetected until next deploy | | Rollback method | git revert | Re-run pipeline with old version | | Audit trail | Git log | Pipeline logs (often ephemeral) | | Credential exposure | Cluster-only (no CI secrets) | CI system has deploy credentials |
Repository Structure
Separate application source code from deployment manifests. This is the most critical architectural decision in GitOps.
# Deployment repo (gitops-deploy) -- separate from application source
gitops-deploy/
apps/
backend/
base/
deployment.yaml
service.yaml
kustomization.yaml
overlays/
dev/
kustomization.yaml # image: backend:dev-abc1234
staging/
kustomization.yaml # image: backend:staging-def5678
prod/
kustomization.yaml # image: backend:v1.2.3
patches/
replicas.yaml
hpa.yaml
frontend/
base/
overlays/
infrastructure/
cert-manager/
ingress-nginx/
monitoring/
clusters/
dev/
kustomization.yaml # References apps/*/overlays/dev
staging/
kustomization.yaml
prod/
kustomization.yaml| Structure Pattern | When to Use | Trade-offs | |-------------------|------------|------------| | Monorepo (app + deploy) | Small teams, single app | Simple but couples build and deploy | | Separate repos | Multiple teams, compliance needs | Clean separation, more repos to manage | | Branch-per-env | Simple promotion model | Risk of branch divergence, merge conflicts | | Directory-per-env | Kustomize overlays | Recommended: explicit, reviewable, no branch drift |
ArgoCD Application Manifest
ArgoCD watches Git repositories and synchronizes Kubernetes cluster state to match.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: backend-prod
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: production
source:
repoURL: https://github.com/myorg/gitops-deploy.git
targetRevision: main
path: apps/backend/overlays/prod
kustomize:
images:
- backend=ghcr.io/myorg/backend:v1.2.3
destination:
server: https://kubernetes.default.svc
namespace: backend
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Revert manual changes in cluster
allowEmpty: false
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
retry:
limit: 3
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas # Ignore HPA-managed replica countArgoCD ApplicationSet for Multi-Environment
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: backend
namespace: argocd
spec:
generators:
- list:
elements:
- env: dev
cluster: https://dev-cluster.example.com
autoSync: true
- env: staging
cluster: https://staging-cluster.example.com
autoSync: true
- env: prod
cluster: https://prod-cluster.example.com
autoSync: false # Prod requires manual sync
template:
metadata:
name: 'backend-{{env}}'
spec:
project: '{{env}}'
source:
repoURL: https://github.com/myorg/gitops-deploy.git
targetRevision: main
path: 'apps/backend/overlays/{{env}}'
destination:
server: '{{cluster}}'
namespace: backend
syncPolicy:
automated:
prune: '{{autoSync}}'
selfHeal: '{{autoSync}}'Flux Kustomization
Flux uses native Kubernetes custom resources and supports multi-tenancy out of the box.
# Source: Git repository to watch
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: gitops-deploy
namespace: flux-system
spec:
interval: 1m
url: https://github.com/myorg/gitops-deploy.git
ref:
branch: main
secretRef:
name: gitops-deploy-auth
---
# Kustomization: what to deploy from that source
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: backend-prod
namespace: flux-system
spec:
interval: 5m
retryInterval: 2m
timeout: 5m
sourceRef:
kind: GitRepository
name: gitops-deploy
path: ./apps/backend/overlays/prod
prune: true
targetNamespace: backend
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: backend
namespace: backend
dependsOn:
- name: infrastructure
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-config
---
# Image automation: watch for new tags, update Git
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
name: backend
namespace: flux-system
spec:
imageRepositoryRef:
name: backend
policy:
semver:
range: '>=1.0.0'Progressive Delivery with Argo Rollouts
Argo Rollouts extends Kubernetes Deployments with canary releases, blue-green deployments, and automated analysis.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: backend
spec:
replicas: 10
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: backend
image: ghcr.io/myorg/backend:v1.2.3
ports:
- containerPort: 8080
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 512Mi }
strategy:
canary:
canaryService: backend-canary
stableService: backend-stable
trafficRouting:
istio:
virtualServices:
- name: backend-vsvc
routes: [primary]
steps:
- setWeight: 5
- pause: { duration: 2m }
- analysis:
templates:
- templateName: success-rate
- setWeight: 25
- pause: { duration: 5m }
- analysis:
templates:
- templateName: success-rate
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
---
# AnalysisTemplate -- automated success criteria
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
args:
- name: service-name
metrics:
- name: success-rate
interval: 30s
count: 5
successCondition: result[0] >= 0.95
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{serv