Skip to content

Data Team Docs

Majority Data Flows

Majority data flows

Data sources, ingestion, warehouse transformations and downstream deliveries across Majority.

Drag to pan; scroll or use + / − to zoom. Fit shows the whole diagram; Readable size restores the original text size. Use Fullscreen for more space. Repeated nodes refer to the same resource across diagrams.

Sources / partners Airflow / dbt processing Storage / BigQuery Messaging / queues Functions / Lambda Reports / destinations Reliability / recovery

Sources and ingestion

flowchart TB
    subgraph n1_27["Sources"]
        n1_3["Bank and payment files: WebBank,<br/>Lithic, CFSB, Checkout, InComm"]
        n1_6["Azure Blob: Majority ledger and<br/>Langfuse"]
        n1_9["APIs: Intercom, Sardine, LoanPro,<br/>Prism, GitHub, Bing, Plausible"]
        n1_12["Rebtel S3: calling and MTU<br/>reconciliation"]
        n1_15["Stitch: QuickBooks and Intercom<br/>history"]
        n1_18["Funnel"]
        n1_21["Google Sheets and CSV seeds"]
        n1_24["Legacy S3 files: partner reports,<br/>Braze and Adjust"]
    end
    subgraph n1_46["Ingestion"]
        n1_28["Airflow extraction jobs on Azure<br/>AKS"]
        n1_31["GCS landing buckets"]
        n1_34["Pub/Sub file-upload topic"]
        n1_37["Ingestion Cloud Function"]
        n1_40["Airflow Rebtel import to Majority<br/>S3"]
        n1_43["S3 ingestion Lambda<br/>s3_lambda"]
        n5_235["Amazon SNS<br/>One topic per bucket"]
    end
    subgraph n1_68["BigQuery and dbt: dw-prod-gwiiag"]
        n1_47["Raw datasets: gcs, s3, kinesis,<br/>event_hub and SaaS"]
        n1_50["dbt staging"]
    end
    n1_28 --> n1_31
    n1_31 -->|"Object finalized"| n1_34
    n1_34 --> n1_37
    n5_235 -->|"S3 event"| n1_43
    n1_40 -->|"S3 notification"| n5_235
    n1_24 -->|"S3 notification"| n5_235
    n1_3 --> n1_28
    n1_6 --> n1_28
    n1_9 --> n1_28
    n1_37 -->|"Load files"| n1_47
    n1_12 --> n1_40
    n1_43 -->|"s3 dataset"| n1_47
    n1_15 --> n1_47
    n1_18 --> n1_47
    n1_21 -->|"External tables and seeds"| n1_50
    class n1_12,n1_15,n1_18,n1_21,n1_24,n1_3,n1_6,n1_9 flow-sources
    class n1_28,n1_40,n1_50 flow-processing
    class n1_31,n1_47 flow-storage
    class n1_34,n5_235 flow-messaging
    class n1_37,n1_43 flow-function

Hold "Alt" / "Option" to enable pan & zoom

See File ingestion and API ingestion for implementation details.

Warehouse transformations

flowchart TB
    subgraph n1_68["BigQuery and dbt: dw-prod-gwiiag"]
        n1_47["Raw datasets: gcs, s3, kinesis,<br/>event_hub and SaaS"]
        n1_50["dbt staging"]
        n1_53["dbt core, intermediate, facts and<br/>dimensions"]
        n1_56["Semantic models"]
        n1_59["Reporting and finance models"]
        n1_62["Reverse-ETL delta models"]
        n1_65["ML features"]
    end
    n1_47 --> n1_50
    n1_50 --> n1_53
    n1_53 --> n1_56
    n1_53 --> n1_59
    n1_53 --> n1_62
    n1_53 --> n1_65
    class n1_50,n1_53,n1_56,n1_59,n1_62,n1_65 flow-processing
    class n1_47 flow-storage

Hold "Alt" / "Option" to enable pan & zoom

Analysis and data products

Read from the BigQuery / dbt models at the top down to each consumer. The ML branch writes inference results back to BigQuery and then into dbt staging.

flowchart TB
    subgraph n1_68["BigQuery and dbt: dw-prod-gwiiag"]
        n1_56["Semantic models"]
        n1_59["Reporting and finance models"]
        n1_62["Reverse-ETL delta models"]
        n1_65["ML features"]
    end
    n1_69["Looker and Count"]
    n1_72["Airflow user-attribute exports"]
    n1_75["Braze"]
    n1_78["Intercom"]
    n1_81["Airflow reporting jobs"]
    n1_84["GCS reports and partner SFTP:<br/>WebBank and Axiom"]
    n1_87["Airflow Sardine exports: hourly"]
    n1_90["Cloud Tasks<br/>Regular + backfill queues"]
    n1_93["Sardine Cloud Function"]
    n1_96["Sardine API"]
    n1_99["Airflow ML inference: fraud, risk,<br/>churn, chargeback"]
    n1_102["BigQuery inference logs"]
    n1_50["Back to dbt staging<br/>BigQuery: dw-prod-gwiiag"]
    n1_72 --> n1_75
    n1_72 --> n1_78
    n1_81 --> n1_84
    n1_87 --> n1_90
    n1_90 --> n1_93
    n1_93 --> n1_96
    n1_99 --> n1_102
    n1_56 --> n1_69
    n1_62 --> n1_72
    n1_59 --> n1_81
    n1_59 --> n1_87
    n1_65 --> n1_99
    n1_102 -->|"Source for downstream models"| n1_50
    class n1_50,n1_56,n1_59,n1_62,n1_65,n1_72,n1_81,n1_87,n1_99 flow-processing
    class n1_102 flow-storage
    class n1_90 flow-messaging
    class n1_93 flow-function
    class n1_69,n1_75,n1_78,n1_84,n1_96 flow-destination

Hold "Alt" / "Option" to enable pan & zoom

Operational and AWS events

Read each flow left to right. Matching topic names connect the diagrams; the historical route is separated from the active flows.

AWS backend event ingestion

Kinesis delivers backend events to a Lambda, which writes to BigQuery and publishes to the warehouse topic. That topic continues in Conversion delivery below.

flowchart LR
    n5_223["BE events<br/>Kinesis: ks-event-prod"]
    n5_226["Backend events Lambda<br/>backend_events_lambda"]
    n1_47["Raw datasets: gcs, s3, kinesis,<br/>event_hub and SaaS"]
    n1_118["Warehouse project: events-topic"]
    n5_223 -->|"Kinesis records"| n5_226
    n5_226 -->|"kinesis.backend_events"| n1_47
    n5_226 -->|"backend_prod events"| n1_118
    class n1_47 flow-storage
    class n1_118,n5_223 flow-messaging
    class n5_226 flow-function

Hold "Alt" / "Option" to enable pan & zoom

Operational event delivery

Service producers and the Airflow publisher feed the event-hub topic. Its conversion branch continues in Conversion delivery below.

flowchart LR
    n1_62["Reverse-ETL delta models"]
    n1_109["Airflow internal-event publisher"]
    n1_106["Service producers: Auditing,<br/>Themis, Simon, WhatsApp"]
    n1_112["Event-hub project: events-topic"]
    n1_115["Backend, Auditing and Themis<br/>standard / batch subscriptions"]
    n1_47["Raw datasets: gcs, s3, kinesis,<br/>event_hub and SaaS"]
    n1_62 --> n1_109
    n1_109 --> n1_112
    n1_106 --> n1_112
    n1_112 --> n1_115
    n1_112 -->|"Native BigQuery subscription"| n1_47
    class n1_106 flow-sources
    class n1_109,n1_62 flow-processing
    class n1_47 flow-storage
    class n1_112,n1_115 flow-messaging

Hold "Alt" / "Option" to enable pan & zoom

The native subscription writes to dw-prod-gwiiag.event_hub.event_log. Backend and Auditing share events-backend-dlq-topic; Themis standard and batch share events-themis-dlq-topic; the BigQuery subscription uses events-bq-dlq-topic. Each has a matching pull subscription, approximately 5 delivery attempts and 7-day dead-letter retention. Backend/Themis dead-letter arrival alerts go to #minority-ops; BigQuery alerts go to #dt-event-hub-dlq. Event Hub configuration.

Conversion delivery

The warehouse topic feeds Adjust. The Event Hub topic feeds the separate web conversion functions with backend and WhatsApp events.

flowchart LR
    warehouse["Warehouse events-topic"] -->|"Selected backend_prod events"| adjust["adjust-api"]
    adjust --> mobile["Adjust S2S API"]
    eventhub["Event Hub events-topic"] -->|"Backend + WhatsApp events"| web["tiktok-api / meta-api / google-ads-api<br/>Separate push subscriptions"]
    web --> destinations["TikTok / Meta / Google Ads"]
    class warehouse,eventhub flow-messaging
    class adjust,web flow-function
    class mobile,destinations flow-destination

Hold "Alt" / "Option" to enable pan & zoom

Event Hub also feeds a braze-api Cloud Run worker pool through separate Auditing and WhatsApp form subscriptions. See Braze event delivery for its retry and dead-letter flow; this is separate from the scheduled Airflow exports above.

Historical mParticle path

This is an inactive reference path: the Kinesis stream was removed and the retained Lambda has no stream mapping. Dashed arrows preserve the historical connections.

flowchart LR
    n5_229["Historical mParticle path<br/>Kinesis stream removed"]
    n5_232["Retained mParticle Lambda<br/>No stream mapping"]
    n1_47["Raw datasets: gcs, s3, kinesis,<br/>event_hub and SaaS"]
    n1_118["Warehouse project: events-topic"]
    n5_229 -.->|"Kinesis records"| n5_232
    n5_232 -.->|"kinesis.mparticle_events"| n1_47
    n5_232 -.->|"mparticle_prod events"| n1_118
    class n1_47 flow-storage
    class n1_118,n5_229 flow-messaging
    class n5_232 flow-function

Hold "Alt" / "Option" to enable pan & zoom

See Ingestion, retry and conversion details for the CDP conversion paths.

Orchestration and reliability

flowchart TB
    subgraph n1_46["Ingestion"]
        n1_37["Ingestion Cloud Function"]
    end
    subgraph n1_68["BigQuery and dbt: dw-prod-gwiiag"]
        n1_50["dbt staging"]
    end
    subgraph n1_140["Orchestration and reliability"]
        n1_128["Airflow on Azure AKS: dbt every 30<br/>minutes and nightly"]
        n1_131["dbt tests and freshness; Slack<br/>failure alerts"]
        n1_134["Ingestion dead-letter queue and<br/>Cloud Run reprocessing"]
    end
    n1_128 --> n1_131
    n1_128 -->|"Schedules"| n1_50
    n1_37 -->|"Failed delivery"| n1_134
    class n1_128,n1_50 flow-processing
    class n1_37 flow-function
    class n1_131,n1_134 flow-reliability

Hold "Alt" / "Option" to enable pan & zoom

Ingestion, retry and conversion details

Detailed flows:

AWS ingestion and retries: source feeds, backend events, failure capture, operator-enabled replay and alerts.
GCP ingestion and retries: collection, loading, automatic retries and manual dead-letter replay.
CDP attribution and conversions: device identity, shared state, mobile conversions and web conversions.

File pickup and delivery schedules

Schedule semantics

All times are UTC. These are configured job starts; actual pickup and delivery depend on processing and dependencies. They are not guaranteed arrival times.

Incoming files

Bank files — SFTP: WebBank bank files and drawdown requests, daily 14:00; CFSB transaction reconciliation, daily 09:15.
Lithic files — SFTP: accounts, cards, transactions and settlement reports, daily 13:00.
Checkout files — SFTP: financial actions, fees and payouts at 00:30, 04:30, 08:30, 12:30, 16:30 and 20:30.
InComm files — SFTP: Cashtie billing and swipe reports, daily 09:30.
Fiserv files — S3: authorizations, chargebacks, fundings and settlements, triggered on upload; arrival time is not configured.
Allpoint files — SFTP: ATM files, Sundays 19:00.
Rebtel reconciliation — S3: calling and mobile top-ups, daily 06:50.
Kard reconciliation — S3: monthly file checks at 07:10 on days 5–21.
Dwolla transfers — S3: daily 10:30.
Remittance provider files — Remitee, GMT, Thunes, Italcambio and Uniteller feeds, daily 15:00.

Platform processing

Airflow and SNS / Lambda handle file ingestion. BigQuery and dbt prepare report data. GCS / S3 retain file copies.

Outgoing files

Axiom compliance — SFTP: check deposits, CIP, overdrafts, balances and ACH; daily send job 09:00.
Axiom settlement — SFTP: posted transactions, roll-forward, settlement and balances; daily send job 13:00.
Axiom fees — SFTP: monthly, 09:00 when the report date is the 2nd of the month.
WebBank reports — SFTP: deposit / credit transactions, positions, reconciliation, funding, loan positions, applications and limit checks; daily report send job 13:00.
WebBank waterfall — SFTP: Tuesdays 13:00.
WebBank funding — SFTP: hourly 12:00–23:00, until that date is uploaded.
CFSB reports — SFTP: daily remittance reports; send job 09:10.
Majority reporting files — GCS: cards / accounts, customers, settled transactions and authorizations; daily generation job 10:00, retained in Majority storage.