GCP Data Engineering · Chapter 02

Big Data Capabilities on GCP

GCP has hundreds of services. This chapter narrows them down: how the cloud changes the job, three ways of using a service, which services matter to a data engineer, and the two ideas everyone trips over first: accounts and quotas.

0 of – checks answered

What the cloud really changes

The definition is simple: renting someone else's server. The consequences for data engineering are not.

	Before the cloud	On the cloud
Getting a server	Order hardware, wait for shipping, cabling and installation	A few clicks or one API call
Time to first table	Months in the book’s on-premises example	Under a minute to load a file into BigQuery
Storage and compute	Bought together with the machine	Services you control with code, on demand
Idle capacity	You paid for it either way	Delete it and stop paying
Test environments	Effectively a second permanent machine	Create for the test, delete afterwards

The data engineering principles stay the same. The technology around them becomes elastic. The clearest example is the ephemeral cluster: create a Spark cluster for one job, run it, delete it, and pay only while it existed.

Three ways to use a service

Every GCP service sits somewhere on a scale from 'you do everything' to 'you do nothing'. Explore each tier, then place some scenarios.

You rent the machine, you run the software

Google gives you a virtual machine (Compute Engine). You install the operating system packages, the software and its updates, and you handle scaling and availability yourself.

You manage: OS packages, software install, scaling, availability, logging
Example: Hadoop installed by hand on Compute Engine VMs; Elasticsearch on VMs
Choose it when: No managed version exists, or you need a very specific version or setup
Trade-off: Most control, most work

Place each scenario

01 You need Elasticsearch, and Google offers no managed version of it.
02 You want Spark jobs to run without installing Hadoop, but you still want to choose the cluster size and network.
03 A 1 TB CSV must be queryable as a table with SQL, and you want zero infrastructure to set up.
04 Your workload needs one exact older version of a tool and custom OS-level packages.

The data engineer's service map

Instead of learning every product, group them by job. Each category has one or two services worth learning first.

Store, process and move large volumes

The core processing services. Most of the book is built on these four.

BigQuery: Fully managed data warehouse
Dataproc: Managed Hadoop: HDFS, MapReduce, Spark, Presto
Dataflow: Fully managed distributed processing, especially good for streaming
Pub/Sub: Fully managed messaging: publish and subscribe to events

Three rules of thumb for choosing: prefer serverless, map the service to where it sits in the data life cycle from Chapter 1, and when two services do the same job, prefer the more widely adopted one, because support, community and hiring are all better.

Accounts and projects

Two identity ideas and one organising idea. Get these right early and later chapters are easier.

	What it is	Typical use
User account	A human’s own login, for example an email address at your company	Console work, ad-hoc queries, development
Service account	An identity for a program, for example a Composer service account	Scheduled jobs, builds, scripts and other automation
Project name	A friendly label. Need not be unique.	Reading the console
Project ID	Must be unique across all of Google Cloud	Commands, APIs and configuration

Which identity should do the work?

01 A Cloud Composer job loads files from Cloud Storage into BigQuery every night.
02 An analyst runs an ad-hoc query in the BigQuery console.
03 A build pipeline creates a Docker image and pushes it whenever code changes.

Using a service account for automation means the company can limit what individual employees see while automated jobs still run. How many projects to create, and when, is Chapter 9.

Quotas versus cost

Money is not the only boundary in the cloud. Quotas are default limits on each service. Decide which kind of limit each situation is.

01 A table design needs more than 10,000 columns in BigQuery.
02 A Cloud Composer environment was left running all month and the bill is high.
03 You try to store a single 6 TB object in Cloud Storage.
04 Your monthly BigQuery queries pass the free tier and charges begin.

Quotas exist for two reasons: services are shared, so no single customer should be able to hurt the rest, and hitting one often signals a design worth reviewing. You do not memorise them; you just know where to look: the quotas page of each service's documentation.

Free tiers cap what is free, not what is possible, and each service prices differently. BigQuery had a free monthly allowance; Cloud Composer bills whenever an environment runs. Chapter 10 covers cost properly.

Hands-on: your first project

Do this in your own Google account. Tick each step as you go; your ticks are saved in this browser.

Register for Google Cloud
Sign in with a Google account at the Cloud console and complete registration. It asks for a payment method. New customers have received trial credit, but check the current free-trial terms first.
Set a budget alert before anything else
In Billing, create a budget with an email alert at a small amount. This step is my addition, not the book’s, and it is the cheapest insurance you will ever buy.
Pin the services you will use
In the navigation menu, pin BigQuery, IAM & Admin, Cloud Storage, Composer, Dataproc, Pub/Sub and Dataflow, so the long service list stops being overwhelming.
Create your own project
From the project menu choose New Project. Pick a unique project ID (a short prefix of your name plus the purpose works well) and then switch into it.
Open Cloud Shell and confirm your project
Cloud Shell is a browser-based Linux terminal with the gcloud command preinstalled.
```
						gcloud config list project
					
```
Run a Python script from the Cloud Editor
Open the editor, create hello_world.py containing print("Hello world"), then run it from the terminal.
```
						python hello_world.py
					
```

Console layouts change, so follow the intent of each step rather than the exact button positions. Nothing here should cost money, but the budget alert protects you if you wander later.

Key terms

Serverless: A fully managed service you simply use, with no infrastructure to provision.
Managed service: Google maintains the software; you still choose sizes and settings.
Compute Engine: Google’s virtual machines, used for the VM-based approach.
Ephemeral cluster: A cluster created for one job and deleted afterwards.
Project: The container for resources, permissions and billing in GCP.
User account: A person’s own identity.
Service account: An identity for non-human work such as automation.
Quota: A default per-service limit, separate from cost.
Cloud Shell: A browser-based Linux terminal with gcloud preinstalled.

Check yourself

Pick an answer to see the reasoning straight away.

A fully managed (serverless) service exists and fits your need. What is the sensible default?

What is the difference between a project name and a project ID?

Why is it common on GCP to create a Spark cluster per job and delete it afterwards?

Which service is the managed version of Apache Airflow?

Why does Google set quotas on its services?

Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.