Skip to content

Google Cloud Platform (GCP)

Google Cloud is our main cloud provider for data workloads, hosting our BigQuery data warehouse.

Overview

  • Project: dw-prod-gwiiag
  • Primary Service: BigQuery (Data Warehouse)
  • Authentication: Majority Google Account

Web Console Access

You can access the GCP console and services:

Log in using your Majority Google account.

CLI Setup

Install Google Cloud SDK

Install the GCloud CLI using Homebrew:

brew install --cask google-cloud-sdk

Authenticate

Authenticate with Google Cloud:

gcloud auth application-default login

Verify Access

Test your access by listing storage buckets in the project:

gcloud storage buckets list

You should see a list of buckets in the dw-prod-gwiiag project.

Reprocessing DLQ Messages

To reprocess Dead Letter Queue (DLQ) messages, refer to the README in our DLQ Cloud Run Job repository:

dt-dlq-cloud-run-job

Adding Bucket to GCS and Table for newly ingested data

[TODO] add it as a separate process documentation

To add new files to GCS (Google Cloud Storage) and create corresponding BigQuery tables:

1. Add Bucket and Notifications

In the dt-gcp-infrastructure repository, add: - The bucket configuration - Its corresponding notifications

2. Add Table Schema

In the dt-gcp-bq-ingestion-cloud-function repository, add: - Schema of the table - The file path for ingestion

Key Rotation

Every 90 days we rotate the keys of our GCP Service Accounts and update their references across all systems that use them for reading from or writing to BigQuery.

Count.Co

  1. Uncomment the service account key generation in Tofu configuration
  2. Run tofu apply to generate the key file locally
  3. Verify in GCP Console: - Navigate to IAM & Admin → Service Accounts → count service account - Go to Manage keys and confirm the new key creation date
  4. In the Count.Co UI: - Go to Settings → DW PROD GCP → Manage connection - Upload the generated JSON file

Update GitHub Secrets

Update the dbt repository secret:

  1. Go to dt-dbt → Settings → Secrets and variables → Repository secrets
  2. Update BIGQUERY_SA_KEY_JSON with the new key value

Data Hub

  1. Uncomment the service account key generation in Tofu configuration
  2. Run tofu apply to generate the key file locally
  3. Verify the key was created in GCP Console
  4. In Data Hub: - Go to Data sources → BigQuery → Configure Connection - Update the private key field with the new value - Update the private key id

N8n

  1. Uncomment the service account key generation in Tofu configuration
  2. Run tofu apply to generate the key file locally
  3. Verify the key was created in GCP Console
  4. Share the generated key file with the Platform team to update at: - n8n.minority.com

Platform Team Access Required

Only the Platform team has access to update n8n credentials.

Airflow

Update the reverse-etl-key in the Azure Key Vault prod-dt-airflow-kv:

  1. Generate the key using Tofu (as described above)
  2. Navigate to Azure Key Vault → prod-dt-airflow-kv
  3. Update the reverse-etl-key secret with the new key JSON

Note

This is required for Airflow DAGs that still use this authentication method instead of impersonation.

Stitch (QuickBooks)

  1. Generate the key using Tofu (as described above)
  2. Log in to Stitch
  3. Navigate to Destinations → BigQuery
  4. Click Upload new key
  5. Upload the generated JSON key file

Looker

  1. Log in to Looker as an admin
  2. Go to Admin → Connections → bq_majority (connection name)
  3. Click Edit
  4. Under Database Settings → Authentication Settings: - Keep Service Account selected - Upload the JSON file
  5. Click Next and Save

AWS Secrets (Kinesis & Lambda)

These secrets are used by Kinesis and Lambda functions. You need to update the secret versions in both GCP and AWS.

Version Numbers

Make sure to increment the version number (e.g., v5 → v6) in the Tofu code.

Plan Changes (example)

tofu plan -target='module.data_warehouse_service_accounts.google_service_account.sa_aws_lambda_kinesis' \
          -target='module.data_warehouse_service_accounts.google_service_account.sa_aws_lambda_s3' \
          -target='module.data_warehouse_service_accounts.google_service_account_key.aws_service_accounts_keys_v5' \
          -target='module.data_warehouse_service_accounts.google_secret_manager_secret.aws_service_accounts_secrets' \
          -target='module.data_warehouse_service_accounts.google_secret_manager_secret_version.version' \
          -target='module.data_warehouse_service_accounts.aws_secretsmanager_secret.secrets' \
          -target='module.data_warehouse_service_accounts.aws_secretsmanager_secret_version.version'

Apply Changes (example)

tofu apply -auto-approve \
          -target='module.data_warehouse_service_accounts.google_service_account.sa_aws_lambda_kinesis' \
          -target='module.data_warehouse_service_accounts.google_service_account.sa_aws_lambda_s3' \
          -target='module.data_warehouse_service_accounts.google_service_account_key.aws_service_accounts_keys_v5' \
          -target='module.data_warehouse_service_accounts.google_secret_manager_secret.aws_service_accounts_secrets' \
          -target='module.data_warehouse_service_accounts.google_secret_manager_secret_version.version' \
          -target='module.data_warehouse_service_accounts.aws_secretsmanager_secret.secrets' \
          -target='module.data_warehouse_service_accounts.aws_secretsmanager_secret_version.version'

Test in Dev First

You can test this process in the dev environment to verify it works before applying to prod.

Verify in AWS

  1. Go to AWS Secrets Manager → Secrets
  2. Click on the secret name
  3. View Secret versions - the current version should show today's date
  4. Verify in GCP Console: - Navigate to the service account under IAM & Admin → Service Accounts - Check the key creation date under Manage keys

Data API Ingestion

These service accounts are located in the data-api GCP project (both dev and prod).

  1. Update the key versions in both dev and prod main.tf files in the Tofu configuration
  2. Run tofu apply for both environments
  3. Update the Azure Key Vault prod-dt-airflow-kv: - Update the airflow-variables-firestore-writer-key secret with the new key value

Final Steps

After updating all systems:

  1. Create a PR in the Tofu repository with all the key rotation changes
  2. Comment out the Tofu code sections that generate local JSON key files
  3. Merge the PR
  4. Verify the GitHub pipeline shows an empty prod plan (since you already applied changes locally)

Clean Up

Delete all locally generated key files after completing the rotation process.