Domain 1

Data Ingestion

Designing and building reliable pipelines to move data from source systems into GCP storage and analytics services.

Cloud Pub/Sub

Pub/Sub is GCP's fully managed, real-time messaging service. It decouples producers from consumers and guarantees at-least-once delivery. Key exam concepts:

Core Concepts

Exam tip: If a question mentions "replay historical messages" or "re-process from a specific point in time," the answer likely involves Pub/Sub seek or snapshots.

When to Use Pub/Sub

Dataflow for Ingestion

While Dataflow is primarily a processing engine, it's often the ingestion layer too — reading from Pub/Sub, Kafka, files, or databases and writing to BigQuery, Bigtable, or GCS.

Batch vs. Streaming Patterns

Cost consideration: BigQuery streaming inserts cost ~$0.01/200 KB. For high-volume ingestion, writing to GCS first and using load jobs (free) can reduce costs dramatically.

Other Ingestion Services

Datastream (CDC)

Serverless change data capture (CDC) service. Reads from MySQL, PostgreSQL, Oracle, and AlloyDB, and streams changes into BigQuery, GCS, or Cloud SQL. Ideal for database replication and real-time analytics on operational data.

Cloud Data Fusion

Visual, code-free ETL/ELT tool built on CDAP. Best for teams that need a GUI-based pipeline builder. Supports 200+ connectors. Runs on Dataproc under the hood.

Storage Transfer Service

Managed service for bulk data transfers from AWS S3, Azure Blob Storage, on-prem filesystems (via agent), or HTTP/HTTPS sources into GCS. Supports scheduled and recurring transfers.

Transfer Appliance

Physical device for migrating petabyte-scale data to GCS when network transfer is impractical. Shipped to your data center, loaded, and shipped back.

Real-time CDCDatastream
Visual ETLCloud Data Fusion
Cloud-to-CloudStorage Transfer Service
Physical migrationTransfer Appliance
Database migrationDatabase Migration Service

Cloud Bigtable — Schema & Ingestion

Bigtable is a high-throughput, low-latency NoSQL database ideal for time-series, IoT, and analytics workloads. Row key design is the single most important decision.

Row Key Design Principles

Exam tip: If a question describes "hotspotting" on a Bigtable table, the answer almost always involves changing the row key — typically by prepending a hash of the primary identifier.

📝 Practice Questions — Data Ingestion

Question 1 — Bigtable Schema Design
You are designing a Cloud Bigtable schema to store IoT sensor readings from 50,000 devices. Each device reports temperature, humidity, and pressure every 10 seconds. Queries will primarily retrieve the last 24 hours of data for a single device, but occasionally need to scan all devices in a specific geographic region for a 1-hour window. The current row key design is deviceId#timestamp. You're observing hotspotting on recently deployed devices that have sequential IDs. How should you redesign the row key?
A Reverse the deviceId string and use reversedDeviceId#timestamp as the row key
B Prepend a hash of the deviceId (e.g., first 2 bytes of MD5) to create hash#deviceId#timestamp, and use a separate Bigtable table with row key region#timestamp#deviceId for regional queries
C Use timestamp#deviceId as the row key so that time-based regional scans are efficient
D Pad the deviceId with leading zeros and add a random salt prefix like salt#deviceId#timestamp
Answer: B. Prepending a deterministic hash of deviceId distributes writes evenly across tablets, eliminating hotspotting while still allowing single-device lookups (since the hash is reproducible). A separate table with region#timestamp#deviceId as the row key efficiently supports the regional scan pattern. Option A (reversing the ID) only partially helps and doesn't address regional queries. Option C (timestamp prefix) creates severe write hotspotting on the most recent timestamp. Option D (random salt) makes reads impossible without scanning all salts.
Question 2 — Streaming Ingestion Cost Optimization
Your team ingests 500 million events per day from mobile apps into BigQuery via Pub/Sub → Dataflow → BigQuery streaming inserts. The data is used for daily batch analytics that run at 6 AM. The streaming insert costs are $15,000/month. Leadership wants to reduce ingestion costs by at least 60% without losing any data. Daily analytics latency of a few hours is acceptable. What should you do?
A Switch from Dataflow to a Cloud Function triggered by Pub/Sub to write directly to BigQuery using streaming inserts
B Replace Pub/Sub with a cron-triggered Storage Transfer Service job that moves files from the mobile backend to GCS
C Keep Pub/Sub → Dataflow, but change the Dataflow pipeline to write windowed output files to GCS, then use BigQuery load jobs (scheduled or triggered) to ingest the data
D Enable Pub/Sub BigQuery subscriptions to write messages directly to BigQuery, bypassing Dataflow entirely
Answer: C. BigQuery load jobs are free (no per-byte charge), while streaming inserts cost ~$0.01/200 KB. By having Dataflow write to GCS in windowed files (e.g., every 5-10 minutes) and then loading into BigQuery, you eliminate the streaming insert cost entirely — well over 60% savings. The Pub/Sub → Dataflow layer is retained to ensure reliable, at-least-once processing. Option A still uses streaming inserts. Option B loses the real-time buffering benefit of Pub/Sub. Option D (Pub/Sub BQ subscriptions) still uses streaming inserts internally.
Question 3 — CDC Replication
Your company runs a PostgreSQL database on Cloud SQL that handles transactional order data. The analytics team needs near-real-time access to this data in BigQuery for dashboards. The current process is a nightly pg_dump export to GCS followed by a BigQuery load job, but the 24-hour data staleness is unacceptable. You need data freshness under 5 minutes with minimal impact on the source database. What should you use?
A Set up a Dataflow JDBC connector pipeline that queries the PostgreSQL database every minute for new or updated rows
B Use Datastream to create a CDC stream from the Cloud SQL PostgreSQL instance directly into BigQuery
C Enable Federated Queries in BigQuery to query the Cloud SQL database directly from dashboards
D Increase the pg_dump frequency to every 5 minutes and automate it with Cloud Scheduler
Answer: B. Datastream is GCP's serverless CDC service. It reads the PostgreSQL WAL (write-ahead log) with minimal impact on the source database and can stream changes directly into BigQuery, achieving sub-minute latency. Option A (polling with JDBC) puts load on the source database and can miss deletes. Option C (Federated Queries) hits the production database for every dashboard query, which is a performance and availability risk. Option D doesn't solve the cost and performance issues of full dumps.