API Ingestion

Besides the file-based feeds (SFTP / Azure Blob), several pipelines pull data from vendor APIs — or receive direct vendor uploads — into GCS buckets. Those buckets are wired to the same Pub/Sub notification (gcs-file-upload-topic), so the dt-gcp-bq-ingestion-cloud-function loads the files into BigQuery gcs.* tables exactly like the file feeds.

Providers

Bing Webmaster Tools


Auth	API key (`bing-api-key` Airflow variable)
DAG	`bing_api_download`
Schedule	`0 7 * * *` (daily at 07:00 UTC)
Image	`file-ingestion`
GCS bucket	`bing-api-{env}`

Reports (each loads into the same-named gcs.* table):

Report
`bing_page_stats`
`bing_query_stats`
`bing_daily_traffic`

Intercom


Auth	API token (`intercom-auth-token` Airflow variable)
DAG	`intercom_api_export`
Schedule	`30 * * * *` (hourly at :30)
Image	`dt-intercom-api/api-export`
GCS bucket	`intercom-api-{env}`

Exports Intercom streams as JSON: admins, tags, contacts, conversations, conversation_parts, calls, call_transcriptions, articles, collections, internal_articles — each into the matching gcs.intercom_* table. See Intercom API for stream/endpoint details.

Sardine


Auth	Client id + secret (`sardine-axiom-client-id` / `sardine-axiom-client-secret` Airflow variables)
DAG	`sardine_api_export`
Schedule	`55 11 * * *` (daily at 11:55 UTC)
Image	`sardine-export`
GCS bucket	`sardine-api-{env}`

Exports alerts, rules, and cases (incremental fetch: wide window, delta filter) into gcs.sardine_alerts / gcs.sardine_rules / gcs.sardine_cases, followed by a dbt step building the stg_sardine_* models.

Separately, the sardine-reconciliation bucket (in the reverse_etl_pipelines project) is written directly by Sardine — their service account has object-creator on the bucket; there is no Airflow DAG. Files load into gcs.sardine_reconciliation_transactions.

Prism


Auth	API token (`prism-api-token` Airflow variable)
DAG	`prism_credit_score`
Schedule	`0 11 1 * *` (monthly, 1st at 11:00 UTC)
Image	`prism-api`
GCS bucket	`prism-api-{env}`

Runs the Prism CashScore evaluation: a dbt model builds the input (rpt_prism_cashscore_evaluation), the Prism API is called, and payloads land under cashscore/ → gcs.prism_cashscore, followed by a dbt build of the stg + mart models.

Blockchain transactions


Auth	API key (`helius-api-key` Airflow variable)
DAG	`blockchain_transactions_ingestion`
Schedule	`10 8 * * *` (daily at 08:10 UTC)
Image	`dt-blockchain-transactions`
GCS bucket	`blockchain-transactions-{env}`

Ingests Solana transactions via the Helius API under solana/ → gcs.blockchain_transactions. Supports backfill via start_time / end_time DAG params.

How it works

flowchart LR
    API["Vendor API"] -->|Airflow DAG task| GCS["GCS Bucket"]
    Vendor["Vendor direct upload"] --> GCS
    GCS -->|PubSub notification| CF["dt-gcp-bq-ingestion\ncloud function"]
    CF --> BQ["BigQuery"]

Hold "Alt" / "Option" to enable pan & zoom

Airflow triggers a KubernetesPodOperatorWithCredentials running a provider-specific image (or the vendor uploads directly)
The task calls the vendor API and uploads the result files to a provider-specific GCS bucket
A PubSub notification (gcs-file-upload-topic) triggers the dt-gcp-bq-ingestion-cloud-function
The cloud function loads the file into BigQuery, adding metadata columns: ingested_at, file_name, bucket_name