GCP Patterns

Build applications and infrastructure using Google Cloud Platform services with appropriate service selection, architecture patterns, and best practices.

Purpose

This skill provides decision frameworks and implementation patterns for Google Cloud Platform (GCP) services across compute, storage, databases, data analytics, machine learning, networking, and security. It guides service selection based on workload requirements and demonstrates production-ready patterns using Terraform, Python SDKs, and gcloud CLI.

When to Use

Use this skill when:

• Selecting GCP compute services (Cloud Run, GKE, Cloud Functions, Compute Engine, App Engine)
• Choosing storage or database services (Cloud Storage, Cloud SQL, Spanner, Firestore, Bigtable, BigQuery)
• Designing data analytics pipelines (BigQuery, Pub/Sub, Dataflow, Dataproc, Composer)
• Implementing ML workflows (Vertex AI, AutoML, pre-trained APIs)
• Architecting network infrastructure (VPC, Load Balancing, CDN, Cloud Armor)
• Setting up IAM, security, and cost optimization
• Migrating from AWS or Azure to GCP
• Building multi-cloud or GCP-first architectures

Core Concepts

GCP Service Categories

Compute Options:

• Cloud Run: Serverless containers for stateless HTTP services (auto-scale to zero)
• GKE (Google Kubernetes Engine): Managed Kubernetes for complex orchestration
• Cloud Functions: Event-driven functions for simple processing
• Compute Engine: Virtual machines for full OS control
• App Engine: Platform-as-a-Service for web applications

Storage & Databases:

• Cloud Storage: Object storage with Standard/Nearline/Coldline/Archive tiers
• Cloud SQL: Managed PostgreSQL/MySQL/SQL Server (up to 96TB)
• Cloud Spanner: Global distributed SQL with 99.999% SLA
• Firestore: NoSQL document database with real-time sync
• Bigtable: Wide-column NoSQL for time-series and IoT (petabyte scale)
• AlloyDB: PostgreSQL-compatible with 4x performance improvement

Data & Analytics:

• BigQuery: Serverless data warehouse (petabyte-scale SQL analytics)
• Pub/Sub: Global messaging and event streaming
• Dataflow: Apache Beam for stream and batch processing
• Dataproc: Managed Spark and Hadoop clusters
• Cloud Composer: Managed Apache Airflow for workflows

AI/ML Services:

• Vertex AI: Unified ML platform (training, deployment, monitoring)
• AutoML: No-code ML for standard tasks
• Pre-trained APIs: Vision, Natural Language, Speech, Translation
• TPUs: Tensor Processing Units for large model training

Decision Framework: Compute Service Selection

Need to run code in GCP?
├─ HTTP service?
│  ├─ YES → Stateless?
│  │  ├─ YES → Cloud Run (auto-scale to zero)
│  │  └─ NO → Need Kubernetes? → GKE | Compute Engine
│  └─ NO (Event-driven)
│     ├─ Simple function? → Cloud Functions
│     └─ Complex orchestration? → GKE | Cloud Run Jobs

Selection Guide:

• First choice: Cloud Run (unless state or Kubernetes required)
• Need Kubernetes: GKE Autopilot (managed) or Standard (full control)
• Simple events: Cloud Functions (60-min max execution)
• Full control: Compute Engine (VMs with custom configuration)

Decision Framework: Database Selection

Choose database type:
├─ Relational (SQL)
│  ├─ Multi-region required? → Cloud Spanner
│  ├─ PostgreSQL + high performance? → AlloyDB
│  └─ Standard RDBMS → Cloud SQL (PostgreSQL/MySQL/SQL Server)
│
├─ Document (NoSQL)
│  ├─ Mobile/web with offline sync? → Firestore
│  └─ Flexible schema, no offline? → MongoDB Atlas (Marketplace)
│
├─ Key-Value
│  ├─ Time-series or IoT data? → Bigtable
│  └─ Caching layer? → Memorystore (Redis/Memcached)
│
└─ Analytics
   └─ Petabyte-scale SQL analytics → BigQuery

Decision Framework: Storage Selection

Storage type needed?
├─ Objects/Files
│  ├─ Frequent access → Cloud Storage (Standard)
│  ├─ Monthly access → Cloud Storage (Nearline)
│  ├─ Quarterly access → Cloud Storage (Coldline)
│  └─ Yearly access → Cloud Storage (Archive)
│
├─ Block storage → Persistent Disk (SSD/Standard/Extreme)
└─ Shared filesystem → Filestore (NFS)

GCP vs AWS vs Azure Service Mapping

Architecture Patterns

Pattern 1: Serverless Web Application

Use Case: Stateless HTTP API with database and caching

Architecture:

Internet → Cloud Load Balancer → Cloud Run → Cloud SQL (PostgreSQL)
                                            → Memorystore (Redis)
                                            → Cloud Storage

Key Services:

• Cloud Run for API service (auto-scaling containers)
• Cloud SQL for transactional data
• Memorystore for caching
• Cloud Storage for file uploads

For detailed Terraform configuration, see references/compute-services.md.

Pattern 2: Data Analytics Platform

Use Case: Real-time event processing and analytics

Architecture:

Data Sources → Pub/Sub → Dataflow → BigQuery → Looker/Tableau
                          ↓
                     Cloud Storage (staging)

Key Services:

• Pub/Sub for event ingestion (at-least-once delivery)
• Dataflow for stream processing (Apache Beam)
• BigQuery for analytics (partitioned tables, clustering)
• Cloud Storage for staging and backups

For BigQuery optimization patterns, see references/data-analytics.md.

Pattern 3: ML Pipeline

Use Case: End-to-end machine learning workflow

Architecture:

Training Data (GCS) → Vertex AI Training → Model Registry → Vertex AI Endpoints
                                                              ↓
                                                         Predictions

Key Services:

• Vertex AI Workbench for notebook development
• Vertex AI Training for custom models (GPU/TPU support)
• Vertex AI Endpoints for model serving (auto-scaling)
• Vertex AI Pipelines for orchestration (Kubeflow)

For ML implementation examples, see references/ml-ai-services.md.

Pattern 4: GKE Microservices Platform

Use Case: Complex orchestration with multiple services

Architecture:

Internet → Cloud Load Balancer → GKE Cluster
                                   ├─ Ingress Controller
                                   ├─ Service Mesh (optional)
                                   ├─ Microservice A
                                   ├─ Microservice B
                                   └─ Microservice C

Key Features:

• GKE Autopilot (fully managed nodes) or Standard (custom configuration)
• Workload Identity for secure GCP service access
• Private cluster with Private Google Access
• Config Connector for managing GCP resources via Kubernetes

For GKE setup and best practices, see references/compute-services.md.

Best Practices

Cost Optimization

Compute:

• Use Committed Use Discounts for predictable workloads (57% off)
• Use Spot VMs for fault-tolerant workloads (60-91% off)
• Cloud Run scales to zero when idle (no charges)
• GKE Autopilot charges only for pod resources, not nodes

Storage:

• Use appropriate Cloud Storage classes (Standard/Nearline/Coldline/Archive)
• Enable Object Lifecycle Management to transition cold data
• Archive backups with Coldline or Archive (99% cheaper than Standard)

Data:

• BigQuery: Use partitioned and clustered tables
• Query only needed columns (avoid SELECT *)