Skip to content

Google Cloud Platform (GCP)

Google Cloud is our main cloud provider for data workloads, hosting our BigQuery data warehouse.

Overview

  • Project: dw-prod-gwiiag
  • Primary Service: BigQuery (Data Warehouse)
  • Authentication: Majority Google Account

Web Console Access

You can access the GCP console and services:

Log in using your Majority Google account.

CLI Setup

Install Google Cloud SDK

Install the GCloud CLI using Homebrew:

brew install --cask google-cloud-sdk

Authenticate

Authenticate with Google Cloud:

gcloud auth application-default login

Verify Access

Test your access by listing storage buckets in the project:

gcloud storage buckets list

You should see a list of buckets in the dw-prod-gwiiag project.

Reprocessing DLQ Messages

To reprocess Dead Letter Queue (DLQ) messages, refer to the README in our DLQ Cloud Run Job repository:

dt-dlq-cloud-run-job

Adding Bucket to GCS and Table for newly ingested data

[TODO] add it as a separate process documentation

To add new files to GCS (Google Cloud Storage) and create corresponding BigQuery tables:

1. Add Bucket and Notifications

In the dt-gcp-infrastructure repository, add: - The bucket configuration - Its corresponding notifications

2. Add Table Schema

In the dt-gcp-bq-ingestion-cloud-function repository, add: - Schema of the table - The file path for ingestion