Q1
GCP Data Engineering · Chapter 06
Streaming with Pub/Sub and Dataflow
Batch pipelines run on a schedule. Streaming pipelines run all the time, handling each event as it arrives. On GCP the pair is Pub/Sub, which carries the messages, and Dataflow, which processes them. This chapter is where time becomes a first-class problem.
0 of – checks answered
01
Batch or streaming?
All data is created in real time. Batching is a simplification we choose because people ask for figures over a period and batches are easier to control. Streaming earns its complexity when the value depends on immediacy.
| Batch | Streaming | |
|---|---|---|
| When it runs | On a schedule (hourly, daily, weekly) | Continuously, one job that never ends |
| Typical question | How much X per day or month? | What is happening right now? |
| Control | You control the schedule and reruns | You handle time, lateness and state |
| Share of pipelines | About 90% (the author’s estimate) | The remainder, but growing |
Which fits?
-
01 The CEO wants last month’s revenue, calculated once from purchase history.
-
02 A card transaction must be checked for fraud the moment it happens.
-
03 A sales dashboard should tick up second by second during a flash sale.
-
04 Daily fact tables are rebuilt each night for the operations team.
The GCP pattern: sources publish to Pub/Sub, Dataflow processes the messages in a distributed way (using Cloud Storage for temporary data if needed), and the results stream into BigQuery. There is no scheduler; the job is long-running.
02
Pub/Sub: a messaging bridge
Pub/Sub receives messages from many systems and hands them to many systems. Play with it: publish messages, pull them with different subscribers, acknowledge them, and let a deadline expire.
Subscription A has two subscribers sharing its load. Subscription B has one. A message published to the topic is copied to both subscriptions.
Nothing published yet.
Grey chips are waiting in the subscription. Blue chips have been delivered but not acknowledged. An ack tells Pub/Sub to stop sending that message; without one it is redelivered until acked or expired.
Name the piece
-
01 The application code that sends each bike-trip JSON message.
-
02 The named store that holds the messages, like a table holds rows.
-
03 An entity that has an interest in a topic. Two of them on one topic each receive identical messages.
-
04 One of several workers that share the load of a single subscription.
Subscription settings that matter
| Setting | What it controls | Trade-off |
|---|---|---|
| Delivery type | Pull (subscriber asks) or push (HTTP endpoint) | Pull is recommended for pipelines; push has a far smaller throughput quota |
| Message retention | How long unacked messages are kept | Longer means less risk of loss but more storage cost |
| Expiration period | When an inactive subscription is deleted | Prevents forgotten subscriptions piling up |
| Ack deadline | How long before an unacked message is resent | Heavy processing needs a longer deadline so subscribers are not overwhelmed |
-
01 Dataflow, using the Pub/Sub client, will read messages for a data pipeline.
-
02 The consumer cannot use the Pub/Sub client and instead exposes an HTTP endpoint.
03
Apache Beam and Dataflow
Dataflow is a serverless processing engine for batch and streaming, positioned like Spark but without a cluster. You write the pipeline with Apache Beam, which hands it to a runner.
Pipe, name, transform
A Beam pipeline is a chain of steps. Each step is written | 'name' >> transform. The name is optional but always worth adding, because Dataflow shows it in the monitoring graph. Beam describes the pipeline; a runner executes it.
- Read
ReadFromText,ReadFromPubSub- Write
WriteToText,WriteToBigQuery- Debug only
Map(print): sample first, never print a big dataset
Beam’s distributed dataset
Reading data gives you a PCollection, the parallel data structure every step consumes and produces. If you know Spark, it plays the role of an RDD.
- Spark equivalent
- RDD
- Each step
- Turns one PCollection into another
- Grouped work
- Combine per key, count per key, windows
One row out, or many
Map returns exactly one output per input. ParDo (Parallel Do) is more flexible: your DoFn class may return one, several or zero rows per input. For a simple split, either works.
- Map
- Exactly one output per input
- ParDo
- A class with a process method; any number out
- Also
- ParDo can read window information
Same code, different engines
DirectRunner runs on your own machine (Cloud Shell), so it starts fast and suits development. DataflowRunner runs on Google’s service, which is right for scale, monitoring and production. You change only the --runner flag.
- Development
- DirectRunner
- Production
- DataflowRunner
- Others exist
- Flink, Samza and more
A small change in Beam
The same framework handles both. Reading a file gives a batch pipeline that finishes. Reading a Pub/Sub subscription with the streaming option turned on gives a pipeline that never ends: it runs until you stop it.
- Batch
- Bounded input, job completes
- Streaming
- Unbounded input, job runs continuously
- Key change
- Read from Pub/Sub and set streaming=True
| Spark on Dataproc | Beam on Dataflow | |
|---|---|---|
| Infrastructure | You create and size a cluster (or make it ephemeral) | Serverless: submit the job |
| Data structure | RDD / DataFrame | PCollection |
| Batch and streaming | Both, with different APIs | Both, with very similar code |
| Billing | Cluster hours | Worker hours; set a maximum number of workers |
Which runner?
-
01 Quickly test a change on the Cloud Shell machine before deploying.
-
02 Run the pipeline in production with autoscaling and a monitoring graph.
A streaming Dataflow job stays on until you stop it. Cancel stops immediately; Drain finishes processing the data already accepted first. The minimum is one worker, so a job left running for a day costs one VM for 24 hours. Cap the maximum workers to avoid surprise autoscaling bills.
04
Windows: giving a stream boundaries
Streams never end, so you cannot sum 'everything'. A window slices time so each slice yields a result. Below is a small stream of bike trips. Change the window size and watch the aggregated rows that would be appended to BigQuery.
| Window start | Station | Trips | Sum of duration (s) |
|---|
In Beam this is WindowInto(FixedWindows(60)), then key by station and CombinePerKey(sum). Each appended row also needs its window
timestamp, otherwise rows for the same station are indistinguishable. A view that sums per station gives you the running total.
05
Hands-on: stream bike trips
Streaming jobs cost money for as long as they run. Do the exercises in one session and cancel everything at the end.
-
Open each service once in the console.
-
Name it informatively, such as
bike-sharing-trips. Untick the default-subscription option so you can create one by hand and see the settings.gcloud pubsub topics create bike-sharing-trips -
Generate a random JSON trip, publish it in a loop of 10, and wait on the publish futures with a callback for errors.
-
Choose pull delivery, a short message retention, an expiration period and an acknowledgement deadline that suits your processing speed.
gcloud pubsub subscriptions create bike-sharing-trips-subs-1 \ --topic=bike-sharing-trips --ack-deadline=60 -
Publish twice, pull in the console, and count 20 messages. Tick "enable ack messages" and ack one so it disappears.
-
Use a venv to avoid install errors, and install the GCP extras. The book pins an old release; use a current supported version.
python3 -m venv beam-env && source beam-env/bin/activate pip install "apache-beam[gcp]" -
Read a text file from Cloud Storage, sample ten lines and print them. Sample first so you never print a huge dataset.
-
Use ParDo to split the log lines, key by URL, count per key and write the result.
-
Change only the runner flag, then open the job graph in the Dataflow console.
-
Read from your subscription with streaming enabled, decode and parse the JSON, and write to a BigQuery table. Test with three tabs: the pipeline, the BigQuery console and the publisher.
-
Use Cancel to stop immediately, or Drain to finish in-flight data first. Dataflow bills by worker hours, and a streaming job will not stop itself.
-
Window, key by start station, combine per key with sum, then add the window timestamp to each record so appended rows are distinguishable.
-
Leave nothing running.
06
Key terms
- Batch / streaming
- Process groups on a schedule, or each event as it arrives.
- Pub/Sub
- Google’s messaging service: publishers, topics, subscriptions, subscribers.
- Topic
- The named store of messages.
- Subscription
- A consumer’s interest in a topic; each gets a full copy.
- Ack
- Acknowledgement that stops redelivery of a message.
- Dataflow
- Serverless data processing for batch and streaming.
- Apache Beam
- The SDK for writing pipelines that runners execute.
- PCollection
- Beam’s distributed dataset.
- ParDo
- A flexible transform that can emit any number of rows per input.
- Runner
- The engine that executes a Beam pipeline, e.g. Direct or Dataflow.
- Window
- A slice of time used to aggregate an unbounded stream.
- CDC
- Change data capture: streaming a database’s inserts, updates and deletes.
07
Check yourself
Pick an answer to see the reasoning straight away.
Q2
A topic has two subscriptions. What does each receive?
Q3
What happens to a message that is delivered but never acknowledged?
Q4
Why aggregate a stream inside time windows?
Q5
A streaming Dataflow job you started is still running after your exercise. What is the consequence?
Q6
How is Dataflow different from Spark on Dataproc, from a user’s point of view?
Study notes in my own words, following the structure of Adi Widjaja's Data Engineering with Google Cloud Platform (Packt, 2022). GCP changes quickly, so check each service's current documentation before relying on names, prices or console steps.