User and Project Management in GCP

Which kind of role?

1. 01 Your own owner role in a throwaway practice project.

   Basic Predefined Custom

2. 02 A data engineer who needs to build BigQuery pipelines, and nothing else.

   Basic Predefined Custom

3. 03 No predefined role has quite the permission set your organisation needs.

   Basic Predefined Custom

Click a node to give the selected person the role there (click again to remove it). Goal: Adam should see everything under the dev folder; Agnes only the marketing project.

Adam can read

Agnes can read

A tall folder tree that grants everything at project level gives you no benefit from folders, only operational overhead. Plan from the bottom up: if some users or service accounts always need the same set of projects, that set deserves a parent folder. There is normally one organization per company.

1. 01 A start-up with a small team building an MVP and no one to run operations.

   One project Split projects

2. 02 A company where the app, data and ML teams use different services and need different roles.

   One project Split projects

3. 03 More than 1,000 active BigQuery users, and a per-project limit of 100 concurrent interactive queries.

   One project Split projects

4. 04 Finance wants to see cost per team on the billing dashboard.

   One project Split projects

Resources in different projects are not isolated: a Composer environment in a data project can read a Cloud SQL database in an application project, as long as the permissions allow it.

Where should the grant go?

1. 01 Data engineers must create and view every table in the warehouse, including tables that do not exist yet.

   Project Dataset Table Column

2. 02 Sales needs the checkout and cart tables. Group those two tables into one dataset and grant on it.

   Project Dataset Table Column

3. 03 One analyst needs to read a single table and nothing else.

   Project Dataset Table Column

4. 04 Marketing may read the user profile table but never its credit card number column.

   Project Dataset Table Column

The e-commerce example in the book: with six tables and four groups, mapping each group to its tables shows which tables cluster together. Create datasets from those clusters and grant roles per dataset. It scales, because a new table dropped into a dataset inherits the access.

Column-level security in three steps

Terraform compares what you declare with what exists. Toggle the dataset declaration, then run the commands in order and watch what each one does.

In main.tf

Actually exists in GCP

Almost every GCP resource can be managed this way. DevOps or infrastructure teams usually own it, but data engineers often contribute the data-related resources, and knowing IaC changes how you design them.

1. List the service accounts in your project

   IAM & Admin, then Service Accounts. Find the default compute service account and any Composer, Dataproc or Cloud SQL ones you created.

2. Find the predefined BigQuery roles

   Read the BigQuery access-control documentation and note which role gives table read access and which lets you run jobs.

3. Plan an ACL on paper

   Take four groups (data engineers, marketing, sales, head of analysts) and six tables. Write down who needs what, then design datasets so each grant covers a whole dataset.

4. Set up column-level security

   Create a taxonomy with a parent tag such as PII and a child tag for credit card numbers, turn on access control enforcement, tag the column in the table schema, and test that an unprivileged user gets an error selecting it.

5. Install Terraform in Cloud Shell

   Download a current release from Terraform’s site, unzip it and move the binary onto your path. Then check the version.

   
   						terraform --version
   					

6. Create backend.tf, main.tf, variables.tf and terraform.tfvars

   The backend stores state in a Cloud Storage bucket, the provider names Google and a project variable, and the tfvars file sets that variable.

   
   						terraform {
     backend "gcs" {
       bucket = "YOUR_BUCKET"
       prefix = "terraform-backend-basic"
     }
   }
   					

7. Run init, plan, apply with nothing declared

   Expect "no changes". You have proven the plumbing works.

   
   						terraform init
   terraform plan
   terraform apply
   					

8. Declare a BigQuery dataset and run plan, then apply

   Read the plan output before typing yes, then check the console.

   
   						resource "google_bigquery_dataset" "new_dataset" {
     project    = var.project_id
     dataset_id = "dataset_from_terraform"
   }
   					

9. Self-exercise: a whole project in code

   A new project, the BigQuery API enabled, three datasets (staging, warehouse, mart) and a service account for BigQuery workloads.

10. Clean up with terraform destroy

    Remove what you created, so nothing lingers.

    
    						terraform destroy
    					

IAM

GCP’s central authorisation service.

Role

A set of permissions you grant to an account.

Basic / predefined / custom

Broad legacy roles; per-service curated roles; roles you define yourself.

Least privilege

Grant only what is needed for the task.

Organization / folder / project

The resource hierarchy. Permissions inherit downward.

ACL

Access control list: IAM applied to data.

Taxonomy / policy tag

A classification scheme and its labels for column-level security.

PII

Personally identifiable information.

IaC

Managing infrastructure through code.

Terraform

Open-source IaC tool using init, plan and apply.