Architecture Overview

How data flows through Majority's platform, from source systems into BigQuery, dbt and the downstream consumers.

flowchart LR
    SFTP["SFTP servers"] -->|dt-file-ingestion| GCS["GCS buckets"]
    API["Vendor APIs"] -->|Airflow DAG tasks| GCS
    LoanPro["LoanPro MySQL"] -->|dt-loanpro| GCS
    GCS -->|PubSub notification| CF["dt-gcp-bq-ingestion\ncloud function"]
    CF --> Raw["BigQuery raw"]
    Raw -->|dbt via Airflow| Models["dbt models"]
    Models --> BI["Looker / Count"]
    Models --> Products["Data products\n(EventHub, file deliveries)"]

Hold "Alt" / "Option" to enable pan & zoom

See Majority data flows for ingestion and event paths, and file pickup and delivery schedules.

Architecture Overview

Details