Pub/Sub

Pub/Sub is GCP's fully managed, real-time messaging service. It decouples producers from consumers and guarantees at-least-once delivery. Key exam concepts:

Core Concepts

• Topics & Subscriptions: Publishers send messages to a topic; subscribers pull (or push) from a subscription attached to that topic. Multiple subscriptions on the same topic enable fan-out.
• Ordering: By default, Pub/Sub does not guarantee ordering. Use ordering_key to enforce ordering within a key partition.
• Exactly-once delivery: Pub/Sub supports exactly-once delivery on the subscription side (pull only). This is distinct from Dataflow's exactly-once processing semantics.
• Dead-letter topics: Messages that can't be processed after N attempts get routed to a dead-letter topic for later inspection.
• Message retention: Default 7 days; configurable up to 31 days. Subscribers can seek/replay to a timestamp or snapshot.

When to Use Pub/Sub

• Streaming event ingestion (clickstream, IoT, logs)
• Decoupling microservices
• Buffering between ingestion and processing (Pub/Sub → Dataflow)
• Fan-out to multiple downstream consumers

Dataflow for Ingestion

While Dataflow is primarily a processing engine, it's often the ingestion layer too — reading from Pub/Sub, Kafka, files, or databases and writing to BigQuery, Bigtable, or GCS.

Batch vs. Streaming Patterns

Click (or tab to) any component below to see why it's used in that pattern instead of the alternative.

Streaming — use for low-latency, continuous data:

1. Buffer Pub/Sub

   Absorbs continuous events with no fixed schedule to batch against, and durably retains them so a downstream outage never loses data.

2. Process Dataflow

   Reads continuously and writes each event through as it arrives — there's no batch to wait for.

3. Warehouse BigQuery (streaming inserts)

   Trade-off: streaming inserts or the Storage Write API make the row queryable within seconds — but cost ~$0.01/200 KB (see the cost note below). That's the price of low latency.

Batch — use for periodic bulk loads where latency tolerance is minutes-to-hours:

1. Source GCS files

   Files already land in GCS on a schedule — there's no continuous stream to buffer, so Pub/Sub adds nothing here.

2. Process Dataflow

   The same Beam pipeline code runs in batch mode — reads the whole file set, transforms it, and finishes, rather than running forever.

3. Warehouse BigQuery (load job)

   Trade-off: load jobs are free — that's the saving the cost note below describes — but you give up the near-instant latency that streaming inserts buy you.

Micro-batch — balances cost and latency:

1. Buffer Pub/Sub

   Same reason as the streaming pattern: events still arrive continuously and need to be absorbed.

2. Window Dataflow (windowed writes)

   Instead of writing each event straight to BigQuery like the streaming pattern does, Dataflow groups events into fixed time-window files first.

3. Stage GCS

   This extra stop is what the streaming pattern skips — it's also what makes the next step a free load job instead of a paid streaming insert.

4. Warehouse BigQuery (load job)

   Trade-off: free, like the batch pattern — but the window is minutes, not hours, so data lands sooner than a full batch run would.

Other Ingestion Services

Datastream (CDC)

Serverless change data capture (CDC) service. Reads from MySQL, PostgreSQL, Oracle, and AlloyDB, and streams changes into BigQuery, GCS, or Cloud SQL. Ideal for database replication and real-time analytics on operational data.

Cloud Data Fusion

Visual, code-free ETL/ELT tool built on CDAP. Best for teams that need a GUI-based pipeline builder. Supports 200+ connectors. Runs on Managed Service for Apache Spark under the hood.

Storage Transfer Service

Managed service for bulk data transfers from AWS S3, Azure Blob Storage, on-prem filesystems (via agent), or HTTP/HTTPS sources into GCS. Supports scheduled and recurring transfers.

Transfer Appliance

Physical device for migrating petabyte-scale data to GCS when network transfer is impractical. Shipped to your data center, loaded, and shipped back.

Bigtable — Schema & Ingestion

Bigtable is a high-throughput, low-latency NoSQL database ideal for time-series, IoT, and analytics workloads. Row key design is the single most important decision.

Row Key Design Principles

• Avoid hotspotting: Sequential keys (timestamps, auto-increment IDs) concentrate reads/writes on a single tablet. Use hashing, salting, or field reversal.
• Support query patterns: Data is stored lexicographically by row key. Design keys so your most common queries are prefix scans.
• Composite keys: Combine fields with # delimiters, e.g. hash#deviceId#timestamp.