Skip to content

GCP ingestion and retries

Source collection, BigQuery loads and dead-letter recovery.

Sources / partners Airflow / dbt processing Storage / BigQuery Messaging / queues Functions / Lambda Reports / destinations Reliability / recovery

Drag to pan; use + / −, Fit or Fullscreen to inspect a flow. Dashed arrows identify retry and replay paths.

Source collection and landing

flowchart LR
    sftp["SFTP file feeds"] --> extraction["dt-file-ingestion<br/>Airflow → pods on Azure AKS"]
    azure["Azure Blob exports"] --> extraction
    api["API adapters"] --> extraction
    extraction -->|"Download / optional decrypt / gunzip"| landing["GCS landing buckets<br/>One source bucket per environment"]
    other["Other producers"] --> landing
    class sftp,azure,api,other flow-sources
    class extraction flow-processing
    class landing flow-storage
Hold "Alt" / "Option" to enable pan & zoom
  • SFTP: WebBank, Lithic credit/debit, CFSB, Checkout, InComm, ATM All Points.
  • Azure Blob: Majority ledger transactions; Langfuse Simon and WhatsApp traces/observations, including observations v2.
  • API adapters: GitHub including coverage, Bing Webmaster, Plausible including nine reports.
  • Other producers: Sardine reconciliation/alerts/rules/cases, Intercom Majority and Rebtel exports, Prism cash score, LoanPro reports, blockchain transactions, Axiom invoices and BIN settlement.

intercom-api-rebtel-prod has its own OBJECT_FINALIZE notification into the shared ingestion topic. sardine-reconciliation in the reverse-ETL project also publishes to that topic. alloy-sftp-prod and talkdesk-call-history-prod are provisioned buckets without ingestion notifications in this configuration; their existence alone does not establish an automatic load path.

dt-file-ingestion supports raw, PGP and BAI2 processing. It selects newer dates and missing files on the latest date, preserves filenames, and uploads under <report>/year=YYYY/month=MM/day=DD/.

Extraction recovery uses an Airflow rerun or source backfill. Individual download errors are logged and skipped; there is no shared file dead-letter queue at this stage.

Event-driven load into BigQuery

flowchart LR
    landing["GCS landing bucket"] -->|"OBJECT_FINALIZE"| topic["Pub/Sub<br/>gcs-file-upload-topic"]
    topic --> subscription["Eventarc subscription<br/>Retry enabled"]
    subscription --> function["ingestion-api<br/>process_file"]
    class landing flow-storage
    class topic,subscription flow-messaging
    class function flow-function
Hold "Alt" / "Option" to enable pan & zoom

The storage notification carries the bucket and object name, not file contents. The Cloud Function Gen2 trigger acknowledges delivery after successful processing. The handler matches the configured bucket and path prefix, skipping excluded or unmapped paths.

Read, normalize and append

flowchart LR
    function["ingestion-api<br/>process_file"] --> read["Read CSV / JSON / JSONL<br/>Load configured schema"]
    read --> temporary["BigQuery temporary table<br/>Normalized field names"]
    temporary -->|"INSERT with SAFE_CAST"| final["BigQuery final table<br/>gcs dataset"]
    class function flow-function
    class read flow-processing
    class temporary,final flow-storage
Hold "Alt" / "Option" to enable pan & zoom
  • Temporary tables expire after one day and are deleted after processing.
  • Final tables are partitioned daily by ingested_at and include ingested_at, file_name and bucket_name metadata.
  • Missing destination schema columns are added.
  • Loads append rows; replay can append duplicates.

Automatic retries and dead-letter capture

flowchart LR
    function["ingestion-api<br/>Processing exception"] -.->|"Function raises error"| retry["Subscription retry policy"]
    retry -.->|"Automatic redelivery"| function
    retry -.->|"About 5 delivery attempts"| topic["Dead-letter topic<br/>gcs-dlq-topic"]
    topic -.-> queue["DLQ pull subscription<br/>gcs-dlq-subscription"]
    class function flow-function
    class retry flow-reliability
    class topic,queue flow-messaging
Hold "Alt" / "Option" to enable pan & zoom

The infrastructure configures a 60–180 second backoff, 120 second acknowledgment deadline and an approximate dead-letter threshold of 5 attempts. The dead-letter topic retains messages for 15 days; the pull subscription retains unacknowledged messages for 15 days.

Manual dead-letter replay

flowchart LR
    queue["DLQ pull subscription<br/>gcs-dlq-subscription"] -.->|"Pull messages"| job["Cloud Run job<br/>gcs-dlq-reprocessing-job"]
    job -.->|"Republish data + attributes"| topic["Original ingestion topic<br/>gcs-file-upload-topic"]
    topic --> processing["Same Eventarc → ingestion-api<br/>→ BigQuery load path"]
    class queue,topic flow-messaging
    class job flow-reliability
    class processing flow-processing
Hold "Alt" / "Option" to enable pan & zoom

An operator runs the job manually. It acknowledges each dead-letter message only after republishing succeeds. The replay job has no task-level retries.

Alerts and sources

  • Function crashes: #dt-cloud-functions-alerts.
  • New dead-letter arrivals (positive percentage change in undelivered messages): #dt-cloud-functions-dlq.
  • Implementation repositories: dt-file-ingestion, dt-gcp-bq-ingestion-cloud-function, dt-gcp-infrastructure, dt-dlq-cloud-run-job, dt-airflow-dags.

Return to data flows.