Rebtel Intercom Export

Purpose

Deliver Rebtel’s Intercom marts from Majority’s BigQuery warehouse for use in Rebtel’s data warehouse.

The Majority-stack intercom_rebtel_api_export pipeline ingests the Rebtel workspace, builds dbt_rebtel, then exports selected marts as full Parquet snapshots.
dt-reporting owns the exporter and manifest contract; the DAG owns the table allowlist.
Output lands in gs://rebtel-intercom-exports-prod in reverse-etl-pipelines-prod, with equivalent -dev resources for Stage Airflow.
dt-gcp-infrastructure manages the buckets and workload-identity access using the shared storage module defaults; no automatic deletion rule is configured.
The current delivery stops at GCS; direct loading into Rebtel’s Snowflake is proposed below and is not yet implemented.

Snapshot folders use snapshots/<UTC-run-timestamp>/<attempt-uuid>/, with the Airflow run start time formatted as 2026-10-20T16-09-05Z; retries keep the timestamp and get a fresh attempt ID.
To find the newest run, choose the greatest timestamp with a completed manifest.json; use completed_at to choose among successful attempts of that run.
Consume only attempts with a completed manifest.json; it lists exact files, row counts and source schemas.
Treat every table as a replacement snapshot, including zero-row tables; deduplicate completed attempts by run_id.
Keep the DAG limited to one active run and avoid independent source rebuilds during export; tables are captured sequentially.
A failed export leaves an incomplete attempt without a manifest. Retry only export_parquet after fixing the error; incomplete attempts remain in GCS until explicitly deleted.
A retry captures current source data under a fresh attempt ID, so clearing an old run does not reconstruct its historical contents.
Deploy bucket/IAM changes first, the dt-reporting image second, and the DAG last; validate a manual stage run before production rollout.

Target flow: BigQuery → GCS → Snowflake, using the existing snapshots without an S3 copy.
In the Rebtel stack: Snowflake storage integration, external Parquet stage, destination tables and scheduled loader
In the Majority stack, grant the integration’s generated Google service account storage.buckets.get, storage.objects.get and storage.objects.list on the export bucket; see the Snowflake integration guide.
The loader discovers completed manifests and uses COPY INTO to load only their listed files into staging tables.
Validate schemas and row counts before replacing destination contents, including clearing tables with zero-row snapshots; retain the previous published snapshot if validation fails.
Record the published run_id and batch_id to make retries idempotent and prevent older runs from replacing newer snapshots.
Schedule manifest-based loading rather than loading each arriving file independently; the manifest is the batch completion signal.