Operations — GCP DE Exam Prep

Cloud Composer (Apache Airflow)

Cloud Composer is GCP's managed Apache Airflow service for workflow orchestration. It is the answer to "how do you schedule and monitor complex multi-step data pipelines?" on the exam.

Core Concepts

DAG (Directed Acyclic Graph): A Python file describing the pipeline — tasks, dependencies, scheduling. DAGs are stored in a GCS bucket and automatically synced to the Airflow scheduler.
Operators: Define what a task does. Key GCP operators: BigQueryInsertJobOperator, DataflowCreatePythonJobOperator, GCSToGCSOperator, DataprocCreateClusterOperator.
Sensors: Pause execution until an external condition is met. GCSObjectExistenceSensor waits for a file to appear in GCS. BigQueryTableExistenceSensor waits for a BQ table.
XComs: Cross-task communication — tasks can push/pull small values between steps. Not suitable for large data (use GCS for that).

Exam tip: Cloud Composer / Airflow is the right answer when you need complex dependency management, retry logic, scheduling, and visibility across multiple GCP services. For simple triggers (e.g., run a Dataflow job when a file arrives in GCS), Cloud Functions or Eventarc may be sufficient.

Composer Environments

Composer 1: GKE-based, requires node pool configuration, more operational overhead.
Composer 2: Autopilot GKE, auto-scaling workers, simpler configuration. Preferred for new deployments.
DAG dependencies (Python packages) are managed via PyPI packages in the environment configuration — no direct worker SSH access needed.

Dataflow — Operational Considerations

Flex Templates

Flex Templates package a Dataflow pipeline as a Docker container stored in Artifact Registry. This allows pipelines to be launched without installing the Apache Beam SDK on the triggering machine. Key advantages over Classic Templates:

Support dynamic parameters resolved at launch time
Can use custom Python/Java packages and Docker base images
Launched via REST API, gcloud CLI, or Cloud Composer
Template stored in GCS (JSON spec) + container in Artifact Registry

Monitoring Dataflow Jobs

System lag: Time between when data was published and when Dataflow processed it. Rising system lag indicates the pipeline is falling behind.
Data freshness: How old the most recently processed watermark is. For streaming jobs, data freshness > acceptable threshold triggers alerts.
Worker CPU / memory: Consistently high CPU suggests need for more workers or larger machine type. Memory pressure can cause OOM kills.
Autoscaling: Dataflow automatically adjusts worker count based on backlog. Set --max_num_workers to cap cost.

Fusion optimization: Dataflow fuses adjacent transforms into a single worker step for efficiency. If a fused step has a bottleneck, use .reshuffle() to break the fusion and allow independent scaling.

Cloud Monitoring & Logging

Cloud Monitoring for Data Pipelines

Custom metrics: Emit application-level metrics (e.g., records processed per second) using the Cloud Monitoring API or OpenTelemetry. Build dashboards and alerting policies on these metrics.
Uptime checks: HTTP/TCP checks to verify service availability. Trigger alerting policies when checks fail.
SLOs & Error Budgets: Define service level objectives on request latency or availability metrics. Monitor error budget burn rate to catch degradation before SLO violations.

Cloud Logging for Data Engineers

Log sinks: Export logs from Cloud Logging to BigQuery, GCS, or Pub/Sub for long-term retention and analysis. Log retention in Cloud Logging is limited (30 days for data access logs by default).
Log-based metrics: Create custom metrics from log patterns (e.g., count of ERROR log entries per minute). Use these to trigger alerting policies.
Audit logs: Data Access audit logs track who read/wrote what data. Critical for compliance. Must be explicitly enabled per-service — they are off by default.

Data Access logsMust enable manually

Admin Activity logsAlways on, 400 days

Log sinkBQ / GCS / Pub/Sub

Log retention30d default, configurable

Cost Management & Attribution

BigQuery Cost Controls

Custom quotas: Set per-user or per-project daily byte limits via the BigQuery IAM quota system. Prevents runaway queries from one analyst exceeding budget.
Cost attribution with labels: Attach labels (team:marketing, env:prod) to BigQuery jobs, Dataflow jobs, and GCS buckets. Export billing data to BigQuery and analyze cost by label.
Committed use discounts: Pre-purchase BigQuery flat-rate slots at a lower effective rate. 1-year and 3-year commitments available.

Dataflow Cost Optimization

Preemptible/Spot VMs: Use --experiments=use_runner_v2 with preemptible workers for batch jobs to reduce compute cost up to 80%. Not recommended for streaming jobs (preemptions cause checkpoint replay).
Horizontal vs. Vertical scaling: Add more workers (horizontal) for parallelizable stages. Increase machine type (vertical) for memory-intensive operations like large window joins.
Idle Dataproc clusters: Use ephemeral clusters (create-on-demand, delete-on-completion via Composer) rather than always-on clusters to avoid idle VM cost.

Exam tip: "How do you allocate BigQuery costs to individual teams?" → Label BigQuery jobs at query time using the labels job configuration field, export billing to BigQuery, and query by label. Alternatively, use separate projects per team with budget alerts.

📝 Practice Questions — Operations

Question 1 — Orchestration Tool Selection

Your data platform runs the following daily pipeline: (1) Wait for an upstream vendor to drop a CSV file in GCS; (2) Run a Dataflow job to clean and transform the data; (3) Load the result into BigQuery; (4) Run a BigQuery stored procedure for aggregations; (5) Send an email notification on success or failure; (6) If step 3 fails, retry up to 3 times with 10-minute intervals. The pipeline has run well in production for 6 months but is hard to debug when failures occur mid-pipeline. What's the best long-term solution?

AReplace the pipeline with a series of Cloud Functions chained via Pub/Sub messages

BMigrate the pipeline to a Cloud Composer DAG using GCS sensors, DataflowOperator, BigQueryOperator, and EmailOperator with built-in retry configuration

CUse Cloud Scheduler + Cloud Run to trigger each step sequentially via HTTP calls

DRe-implement the pipeline as a single Dataflow job that handles all steps internally

Answer: B. Cloud Composer (Airflow) is purpose-built for exactly this type of multi-step pipeline: GCS sensors wait for file arrival, operator-level retry policies handle transient failures, the Airflow UI provides step-level visibility for debugging, and built-in email operators handle notifications. Cloud Functions (A) lack centralized monitoring and complex retry logic is difficult to manage across chained functions. Cloud Scheduler + Cloud Run (C) has no dependency management or retry coordination. Dataflow (D) is a data processing engine, not an orchestrator — it can't natively wait for external files or send emails.

Question 2 — Dataflow Streaming Monitoring

You operate a Dataflow streaming job that processes Pub/Sub messages and writes results to BigQuery. Over the weekend, your on-call engineer receives an alert: the pipeline is processing correctly but analysts report that dashboard data is 45 minutes stale. The Dataflow job shows no errors. What metric should you check first, and what likely caused the issue?

ACheck worker CPU utilization — high CPU caused processing slowdown

BCheck Pub/Sub subscription oldest unacked message age — messages are stuck in the subscription

CCheck Dataflow system lag and data freshness metrics — rising system lag indicates the pipeline is backlogged and processing data behind its watermark

DCheck BigQuery streaming insert error rate — inserts may be failing silently

Answer: C. System lag and data freshness are the primary Dataflow streaming health metrics. System lag measures the gap between event publish time and processing time — a 45-minute staleness maps directly to high system lag. The pipeline is running (no errors) but backlogged, likely due to a traffic spike or worker scaling lagging behind throughput. Pub/Sub oldest unacked age (B) is also relevant but is upstream of Dataflow — Dataflow's own system lag tells you whether the backlog is in Pub/Sub or in Dataflow processing. CPU (A) might be a contributing cause but isn't the primary diagnostic metric for staleness. BigQuery insert errors (D) would cause data loss, not staleness, and would appear as errors in the job graph.

Question 3 — Cost Attribution

Your organization has 8 teams sharing a single GCP project with BigQuery. Finance needs a monthly report showing BigQuery costs broken down by team. Currently there is no way to distinguish which team ran which queries. You need to implement cost attribution with minimal impact on existing team workflows. What should you do?

ARequire all teams to add a team label to their BigQuery jobs (via job configuration or the --label flag in bq CLI); export billing data to BigQuery; query costs by label in the billing dataset

BCreate a separate GCP project for each team and use GCP billing export to separate costs

CUse Cloud Monitoring custom dashboards to track per-user slot usage

DAssign individual service accounts to each team member and track costs by service account in billing export

Answer: A. BigQuery job labels are the standard GCP mechanism for cost attribution within a shared project. Labels are passed through to billing export, enabling SQL queries like SELECT labels.value as team, SUM(cost) FROM billing_table WHERE labels.key='team' GROUP BY 1. This requires minimal workflow change — teams add --label team:marketing to their queries. Separate projects (B) requires data to be duplicated or shared across projects and is a major operational change. Cloud Monitoring (C) tracks slot usage but doesn't map to billing costs. Per-service-account tracking (D) works but requires each team member to authenticate with a shared service account, which is an anti-pattern and complicates security.