Data Processing — GCP DE Exam Prep

Dataflow & Apache Beam

Dataflow is Google's managed runner for Apache Beam pipelines. It handles both batch and streaming with the same programming model. This is the most heavily tested topic on the exam.

Windowing

Fixed windows: Non-overlapping, uniform-size time intervals (e.g., every 1 hour). Best for regular aggregations.
Sliding windows: Overlapping windows defined by size + slide interval (e.g., 1-hour windows every 5 minutes). Best for moving averages.
Session windows: Dynamic windows that close after a gap of inactivity. Best for user sessions.
Global window: Single window containing all data. Requires custom triggers to emit results.

Triggers & Watermarks

Watermark: Beam's estimate of when all data up to time T has arrived. Data before the watermark is "on-time"; data after is "late."
AfterWatermark trigger: Fires when the watermark passes the end of the window — the default for getting "on-time" results.
Late firings: Additional trigger firings for late data that arrives after the watermark. Configured with withLateFirings().
Allowed lateness: How long after the watermark passes a window's end Dataflow keeps state for that window. Late elements outside this window are dropped.
Accumulation mode: ACCUMULATING (each firing includes all data seen so far) vs. DISCARDING (each firing only includes new data since last fire).

Exam tip: The combination of windowing + triggers + allowed lateness + accumulation mode is the most common Dataflow question pattern. Understand each piece independently, then how they compose.

Key Dataflow Features

Autoscaling: Automatically adjusts workers based on backlog. Streaming pipelines use Streaming Engine for more efficient scaling.
Exactly-once processing: Dataflow guarantees exactly-once semantics for both batch and streaming with Streaming Engine.
Shuffle service: Offloads the GroupByKey shuffle to a managed service, reducing worker disk needs. Enabled by default for batch.
Flex Templates: Package pipelines as Docker containers for reusable, parameterized deployment.

BigQuery Optimization

BigQuery is a serverless, petabyte-scale data warehouse. Cost optimization and query performance are common exam topics.

Partitioning & Clustering

Partitioning: Divides a table into segments by a column (typically date/timestamp or integer range). Queries that filter on the partition column only scan relevant partitions — reducing cost and improving speed.
Clustering: Sorts data within each partition by up to 4 columns. Most effective when combined with partitioning. Clustering improves filter and aggregation performance on the clustered columns.
Require partition filter: You can enforce that queries must include a partition filter, preventing accidental full-table scans.

Materialized Views

Pre-computed query results that BigQuery automatically refreshes. Queries against base tables are transparently rewritten to use materialized views when possible (smart tuning). They're ideal for dashboards with repeated aggregation queries.

Cost Controls

On-demand vs. flat-rate (editions): On-demand charges per TB scanned; editions provide dedicated slots with predictable pricing.
Slot reservations: Use for predictable, high-volume workloads. Autoscaling editions adjust slots dynamically.
Storage pricing: Active storage vs. long-term storage (data unmodified for 90+ days gets automatic discount).

Common trap: "Use APPROX_COUNT_DISTINCT to reduce costs" — approximate functions reduce compute time but do not reduce bytes scanned (which determines on-demand cost). The correct cost optimization is usually partitioning, clustering, or materialized views.

Dataproc (Spark / Hadoop)

Managed Spark and Hadoop clusters. Use when you have existing Spark/Hadoop code or need the Spark ecosystem (MLlib, GraphX, etc.).

When Dataproc vs. Dataflow

Dataproc: Existing Spark/Hadoop jobs, need Spark ML or Spark SQL, complex graph processing, teams with Spark expertise.
Dataflow: New pipelines, unified batch+streaming, exactly-once semantics, serverless (no cluster management).

Optimization Tips

Use ephemeral clusters — create per-job, delete after. Cloud Composer can orchestrate this.
Store data in GCS (not HDFS) to decouple storage from compute.
Use preemptible/spot VMs for worker nodes to reduce cost (not for the master).
Use Dataproc Serverless for Spark batch jobs without managing clusters.

Cloud Composer (Airflow)

Managed Apache Airflow for orchestrating complex data pipelines. DAGs define task dependencies and scheduling.

Use for orchestrating multi-step pipelines: ingest → transform → validate → load.
Built-in operators for BigQuery, Dataflow, Dataproc, GCS, and more.
Sensor operators can wait for conditions (file arrival, API readiness) before proceeding.
Composer 2 is preferred — it's built on GKE Autopilot and supports autoscaling.

📝 Practice Questions — Data Processing

Question 1 — Dataflow Streaming & Windowing

Your company processes clickstream data from a global e-commerce platform using a Dataflow streaming pipeline. Events arrive with varying delays — some up to 30 minutes late due to mobile network issues. You need to compute hourly revenue aggregations that are both timely and eventually accurate. The business requires preliminary results within 2 minutes of each hour's end, but also needs updated results as late data arrives. What windowing and triggering strategy should you use?

A Use session windows with a 30-minute gap duration and a single AfterWatermark trigger with no late firings

B Use fixed 1-hour windows with a trigger that fires AfterWatermark for on-time results, plus late firings using AfterProcessingTime.pastFirstElementInPane().plusTimeDuration(5 minutes), and set allowed lateness to 30 minutes

C Use sliding windows of 1 hour with a 2-minute slide, using Repeatedly.forever(AfterPane.elementCountAtLeast(1)) as the trigger

D Use global windows with periodic 1-hour triggers using AfterProcessingTime, and manually discard late elements using a DoFn filter

Answer: B. Fixed 1-hour windows match the hourly aggregation requirement. The AfterWatermark trigger provides timely initial results when the watermark passes the window end. Late firings with AfterProcessingTime provide updated results as late data trickles in. Setting allowed lateness to 30 minutes matches the maximum expected delay. Option A uses session windows (wrong granularity). Option C creates 30 overlapping windows per element (wasteful, and triggers on every element). Option D uses global windows which require manual time management.

Question 2 — BigQuery Cost Optimization & Materialized Views

Your analytics team runs a daily dashboard that joins a 10 TB fact_orders table (partitioned by order_date, clustered by customer_id) with a 500 MB dim_products table. The dashboard filters on order_date within the last 7 days and aggregates revenue by product category. The query costs approximately $50/day. You need to reduce costs by at least 80% while keeping data fresh within 1 hour. What approach best meets these requirements?

A Create a BigQuery materialized view that pre-aggregates revenue by product category and order_date, leveraging automatic refresh and smart tuning so the dashboard queries hit the materialized view instead of the base tables

B Export the last 7 days of data nightly to a Cloud Storage bucket as Parquet files, create an external table over them, and point the dashboard to the external table

C Convert the dashboard query to use APPROX_COUNT_DISTINCT and APPROX_QUANTILES for all aggregations to reduce data scanned

D Schedule a nightly SQL script that writes results to a separate dashboard_summary table, and point the dashboard to that table

Answer: A. Materialized views pre-compute and store aggregation results. BigQuery automatically refreshes them as base data changes and uses "smart tuning" to rewrite queries against the materialized view. This dramatically reduces bytes scanned (from TB to KB/MB) for repeated dashboard queries. Option B loses freshness (nightly export) and external tables are slower. Option C doesn't reduce bytes scanned — approximate functions only reduce compute. Option D provides stale data (nightly refresh) and doesn't meet the 1-hour freshness requirement.

Question 3 — Pipeline Orchestration

Your data platform has a daily pipeline: (1) Extract from 3 source databases, (2) Load raw data into GCS, (3) Run a Dataflow job to clean and transform, (4) Load into BigQuery, (5) Run data quality checks, (6) Trigger downstream ML training only if quality checks pass. Steps 1-2 can run in parallel for all 3 sources. The current implementation uses cron jobs and a bash script that breaks frequently. What should you use?

A Cloud Scheduler triggering Cloud Functions for each step, using Pub/Sub for inter-step communication

B A single Dataflow pipeline that handles all steps including extraction, transformation, loading, quality checks, and ML triggering

C Cloud Composer (Airflow) with a DAG that defines task dependencies, uses parallel task groups for the 3 source extractions, and includes a BranchPythonOperator for conditional ML training based on quality check results

D Cloud Workflows with step-based YAML definitions and retry policies for each service call

Answer: C. Cloud Composer is designed exactly for this pattern — orchestrating multi-step, multi-service data pipelines with complex dependencies. DAGs provide declarative dependency management, parallel task execution, branching logic, and built-in retry/alerting. Option A is fragile and hard to maintain for complex dependencies. Option B conflates orchestration with processing — Dataflow shouldn't handle extraction or ML triggering. Option D (Cloud Workflows) is lighter-weight and better for simple API chains, not complex data pipeline orchestration with branching.