GCP WAF Reliability Review
Purpose
Evaluate GCP workload reliability against the Google Cloud Well-Architected Framework (WAF) reliability pillar. This skill supports consistent performance, resilience to failures, and graceful recovery for workloads running on GCP.
When to use
Use this skill for:
- Assessing availability requirements and SLO definitions for a workload
- Designing highly available, resilient architectures on GCP
- Auditing a workload against the nine WAF reliability principles
- Evaluating redundancy topology, observability coverage, and failure testing maturity
- Reviewing postmortem culture and data recovery practices
WAF Reliability Pillar Overview
The GCP Well-Architected Framework reliability pillar provides guidance across nine core principles. Together they ensure workloads deliver consistent performance to users, remain resilient under fault conditions, and recover gracefully from both partial and complete failures. The pillar emphasizes user-experience-driven SLOs, error budget discipline, horizontal scalability, and continuous improvement through postmortems.
Core Principles
1. Define Reliability Based on User-Experience Goals
Anchor reliability targets to user-observable outcomes rather than internal infrastructure metrics. Define SLIs that reflect what users actually care about — latency, availability, correctness, and throughput.
- Reference: https://cloud.google.com/architecture/framework/reliability/define-reliability-based-on-user-experience-goals
2. Set Realistic Targets for Reliability
Establish SLOs and error budgets that are achievable, meaningful, and aligned with business risk tolerance. Avoid aspirational targets that drain engineering capacity without delivering user value.
- Reference: https://cloud.google.com/architecture/framework/reliability/set-targets
3. Build Highly Available Systems Through Resource Redundancy
Eliminate single points of failure by distributing resources across multiple zones and regions. Design for fault isolation so that the failure of any single component does not cascade to a full system outage.
- Reference: https://cloud.google.com/architecture/framework/reliability/build-highly-available-systems
4. Take Advantage of Horizontal Scalability
Design systems to scale out rather than up. Use managed instance groups, GKE node pools, and serverless platforms to handle load increases without manual intervention and without architectural bottlenecks.
- Reference: https://cloud.google.com/architecture/framework/reliability/horizontal-scalability
5. Detect Potential Failures by Using Observability
Instrument workloads with metrics, logs, and traces that provide actionable signals before failures reach users. Use SLI-based alerting to distinguish user-impacting events from noise.
- Reference: https://cloud.google.com/architecture/framework/reliability/observability
6. Design for Graceful Degradation
Architect systems to maintain partial functionality under failure conditions. Implement circuit breakers, retries with exponential backoff, bulkheads, rate limiting, and fallback paths to prevent cascading failures.
- Reference: https://cloud.google.com/architecture/framework/reliability/graceful-degradation
7. Perform Testing for Recovery from Failures
Validate resilience assumptions through chaos engineering and game days. Test failure modes including zone outages, dependency timeouts, traffic surges, and misconfigurations in pre-production and, where safe, in production.
- Reference: https://cloud.google.com/architecture/framework/reliability/perform-testing-for-recovery-from-failures
8. Perform Testing for Recovery from Data Loss
Validate backup and restore procedures regularly. Test recovery time objective (RTO) and recovery point objective (RPO) compliance under realistic failure scenarios, not just in tabletop exercises.
- Reference: https://cloud.google.com/architecture/framework/reliability/perform-testing-for-recovery-from-data-loss
9. Conduct Thorough Postmortems
Build a blameless postmortem culture to systematically capture incident learnings. Track action items to completion and share findings to prevent recurrence across teams.
- Reference: https://cloud.google.com/architecture/framework/reliability/conduct-postmortems
Relevant GCP Products
- Compute & Containers: Compute Engine Managed Instance Groups (MIGs), Google Kubernetes Engine (GKE), Cloud Run, Cloud Run Functions
- Load Balancing & Networking: Cloud Load Balancing (global and regional), Cloud CDN, Cloud DNS with routing policies, Traffic Director
- Data & Storage: Cloud Storage (multi-region buckets), Cloud SQL with HA and read replicas, Cloud Spanner (multi-region), Bigtable, Firestore
- Observability: Cloud Monitoring, Cloud Logging, Cloud Trace, Google Cloud Managed Service for Prometheus, Error Reporting, SLO monitoring
- Backup & Resilience: Backup and DR Service, Cloud Storage versioning, Cloud SQL automated backups, Spanner PITR
- Autoscaling: MIG autoscaler, GKE Horizontal Pod Autoscaler (HPA), GKE Vertical Pod Autoscaler (VPA), GKE cluster autoscaler, Cloud Run concurrency-based scaling
Assessment Question Bank
SLO/SLI Definition and Error Budgets
- Are SLIs defined based on user-observable outcomes (latency, availability, correctness) rather than infrastructure metrics?
- Is an SLO set for each user-facing service with an associated error budget?
- Is the error budget actively tracked and used to gate feature velocity vs. reliability investment?
- Are SLOs documented in a service-level agreement (SLA) or internal reliability contract?
- Are SLOs reviewed and updated when user expectations or product requirements change?
- Is there a process for alerting when error budget burn rate is too high?
Realistic Reliability Targets
- Were SLO targets derived from user research, business requirements, or dependency constraints rather than set arbitrarily?
- Is the cost of each additional nine (99.9% vs. 99.99%) understood and justified?
- Are dependencies' reliability levels accounted for when setting the workload's own SLO?
- Is there a tiering model distinguishing critical vs. non-critical services with different reliability targets?
- Are error budget policies defined (e.g., freeze new features when budget is exhausted)?
High Availability Topology
- Are workloads deployed across at least two availability zones within a region?
- Are stateful services (databases, caches) configured for multi-zone or multi-region replication?
- Are managed instance groups or GKE deployments configured with anti-affinity rules to spread replicas across zones?
- Is Cloud Load Balancing configured with health checks that remove unhealthy backends automatically?
- Are DNS failover or routing policies used to redirect traffic during regional impairments?
- Is there a documented recovery procedure for a full zone failure?
- Are global load balancers used for latency-sensitive workloads requiring cross-region failover?
Horizontal Scalability
- Are workloads designed to scale out (add instances) rather than scale up (increase instance size)?
- Are autoscaling policies configured and tested for all stateless compute tiers?
- Is there capacity headroom available to absorb unexpected traffic spikes without manual intervention?
- Are stateful components (databases, queues) able to scale horizontally or handle peak load without bottlenecks?
- Are scaling decisions based on meaningful metrics (request latency, queue depth) rather than just CPU utilization?
- Are scale-in operations safe — do they wait for in-flight requests to complete before terminating instances?
Observability
- Are the four golden signals (latency, traffic, errors, saturation) instrumented for each service?
- Are SLI metrics available in Cloud Monitoring with SLO policies configured?
- Is distributed tracing enabled to diagnose cross-