Data Storage — GCP DE Exam Prep

Cloud Storage (GCS)

GCS is GCP's unified object storage service. Understanding storage classes and lifecycle management is heavily tested on the exam.

Storage Classes

Standard: Highest availability, no minimum storage duration. Best for frequently accessed ("hot") data.
Nearline: Low-cost for data accessed at most once per month. 30-day minimum storage duration. ~$0.01/GB/month retrieval.
Coldline: Very low-cost for data accessed at most once per quarter. 90-day minimum. ~$0.02/GB/month retrieval.
Archive: Lowest cost for long-term archival (accessed less than once per year). 365-day minimum. ~$0.05/GB/month retrieval.

Exam tip: When asked which class to use, look at access frequency. Monthly backups → Nearline. Compliance archives → Archive. Retrieval cost penalties matter when data is accessed more than expected — always match class to access pattern.

Lifecycle Management

Lifecycle rules automatically transition objects between classes or delete them after conditions are met. Common conditions: age (days since creation), numNewerVersions, isLive. Common actions: SetStorageClass, Delete.

Watch out: Transitioning to Nearline requires objects to be at least 30 days old. Jumping directly from Standard to Coldline or Archive skips the minimum duration penalty only if you never stored it in an intermediate class.

Retention Policies & Object Lock

Retention policies prevent object deletion before a minimum age. Object Lock (Bucket Lock) enforces WORM (Write Once, Read Many) compliance — once locked, the policy cannot be reduced or removed. Used for regulated data like financial records.

Cloud Bigtable

Bigtable is a fully managed, petabyte-scale NoSQL wide-column store optimized for high-throughput, low-latency workloads. It is not a relational database — there is no SQL support and no cross-row transactions.

When to Use Bigtable

Time-series data (IoT sensor readings, financial tick data)
Analytical workloads requiring millions of reads/writes per second
Data that doesn't require joins or complex queries
Wide rows with hundreds to thousands of columns per row

Row Key Design

Row key design is the most critical Bigtable decision. Data is stored sorted lexicographically by row key and distributed across tablets. Poor key design causes hotspotting.

Avoid: Sequential numeric IDs, timestamps as key prefix (recent writes all hit one tablet)
Use: Hash-prefixed keys, reversed timestamps (MAX_TIMESTAMP - event_time), composite keys that distribute writes
Pattern: hash(entity_id)#entity_id#reverse_timestamp for per-entity time-series with even distribution

Tablet splitsAutomatic at ~1 GB

Column familiesGroup related columns

Garbage collectionPer column family

ReplicationMulti-cluster, async

Cloud SQL

Cloud SQL is GCP's managed relational database service supporting MySQL, PostgreSQL, and SQL Server. It is designed for OLTP workloads, not large-scale analytics.

Key Features for the Exam

High Availability (HA): Synchronous replication to a standby in a different zone. Automatic failover with ~60 second downtime.
Read replicas: Asynchronous cross-region replicas for scaling reads and disaster recovery. Not automatic failover targets.
Automatic storage increase: Cloud SQL can auto-grow storage but cannot shrink it — plan initial sizing carefully.
Private IP + VPC peering: Best practice for production. Cloud SQL Auth Proxy handles encryption and IAM auth without VPN.
Backups: Automated daily backups + point-in-time recovery (PITR) using binary log. PITR requires binary logging to be enabled.

Exam tip: Cloud SQL is for existing RDBMS workloads at moderate scale (up to ~30 TB). For horizontal scale-out, global distribution, or consistency across regions → use Cloud Spanner.

Cloud Spanner

Spanner is GCP's globally distributed, horizontally scalable, strongly consistent relational database. It combines the SQL semantics of a relational database with the horizontal scale of NoSQL.

Core Differentiators

External consistency: Stronger than serializable — every transaction appears to execute at a globally consistent timestamp. Enabled by TrueTime (atomic clocks + GPS).
Automatic sharding: Spanner splits and rebalances data across splits automatically. Row key design still matters to avoid hotspots.
Interleaving: Parent-child table rows stored physically together on disk for efficient joins. Use INTERLEAVE IN PARENT for one-to-many relationships with frequent co-reads.
Stale reads: Read at a past timestamp (e.g., 15 seconds ago) for lower latency and cost. Use when strong consistency is not required.

Anti-patterns to avoid: Monotonically increasing primary keys (e.g., auto-increment integers) cause write hotspots on the last split. Use UUID, hash, or bit-reversed sequences.

Firestore

Firestore is GCP's serverless, horizontally scaling document database. It is designed for mobile/web backends requiring real-time sync and offline support.

Datastore Mode vs. Native Mode

Native mode: Supports real-time listeners, mobile/web SDKs, and offline sync. Cannot use Datastore API.
Datastore mode: Backwards compatible with Cloud Datastore API. Does not support real-time listeners or mobile SDKs. Better for server-side workloads.

Exam tip: Firestore is the go-to answer for "mobile app backend needing offline support and real-time sync." For IoT or time-series analytics → Bigtable. For RDBMS with transactions → Cloud SQL or Spanner.

Choosing the Right Storage Service

Object/filesCloud Storage

Time-series / IoTBigtable

RDBMS < 30 TBCloud SQL

Global RDBMSCloud Spanner

Mobile / offlineFirestore Native

OLAP / warehouseBigQuery

📝 Practice Questions — Data Storage

Question 1 — Storage Class Selection

Your company stores regulatory compliance documents that must be retained for 7 years. These documents are accessed once per year during audits. The documents average 5 MB each and you have 2 million of them. What storage class should you use to minimize cost while meeting the access requirements?

AStandard — high availability ensures documents are always accessible

BNearline — 30-day minimum is acceptable for annual access

CArchive — access less than once per year makes Archive the lowest-cost option; the 365-day minimum storage matches the annual access pattern

DColdline — 90-day minimum is a good balance between cost and retrieval fees

Answer: C. Archive class has the lowest storage cost (~$0.0012/GB/month) and is designed for data accessed less than once per year. The 365-day minimum storage duration aligns with the annual audit access pattern. Coldline (D) would also work technically but costs more per GB to store. Standard (A) and Nearline (B) are significantly more expensive for data touched once a year.

Question 2 — Bigtable Row Key Design

You're designing a Bigtable table for a financial trading platform that stores stock price ticks. The primary access pattern is: "retrieve all ticks for a given stock symbol in the last 5 minutes." You expect 10,000 symbols with highly variable write rates — some symbols generate 1,000 ticks/sec, others only 1/sec. The current row key is symbol#timestamp. You're observing hotspotting on the most active symbols. What's the best row key redesign?

AChange to timestamp#symbol so rows are ordered by time globally

BUse shard_id#symbol#reverse_timestamp where shard_id = hash(symbol) % 10, querying all shards for a symbol's range scan

CUse a UUID as the row key to guarantee even distribution

DPad the symbol with leading zeros to make keys the same length

Answer: B. The hotspot occurs because all writes for a popular symbol concentrate on one Bigtable tablet. Salting with a deterministic hash shard (e.g., shard_id#symbol#reverse_timestamp) distributes writes for hot symbols across multiple tablets. Using reverse_timestamp means the most recent ticks sort first. You query all 10 shards for a symbol in parallel and merge results — a common Bigtable scatter-gather pattern. Option A creates write hotspots on the current timestamp. Option C (UUID) makes range scans by symbol impossible. Option D (padding) doesn't address the distribution problem.

Question 3 — Storage Service Selection

You are building a global e-commerce platform that processes 50,000 transactions per second across 5 regions (North America, Europe, Asia-Pacific, South America, Australia). Each transaction must update inventory counts and order status atomically. The system requires strong consistency — a customer in Tokyo and a customer in London must see the same inventory count immediately after a purchase. You need SQL query support for reporting. Which storage service should you use?

ACloud SQL with multi-region read replicas and a distributed cache layer

BBigtable with a multi-cluster replication configuration

CFirestore in Native mode with multi-region configuration

DCloud Spanner with a multi-region instance configuration

Answer: D. Cloud Spanner is the only GCP service that provides: globally distributed writes, external consistency (stronger than serializable), full SQL support, and automatic horizontal scaling. It is specifically designed for this use case. Cloud SQL (A) can't write globally — read replicas don't support writes, and cross-region writes via a single primary introduces high latency and single points of failure. Bigtable (B) doesn't support SQL or ACID transactions across rows. Firestore (C) supports transactions within a document but is not designed for high-throughput global financial transactions with complex SQL reporting.