Skip to content

Publishing flow

Overview

For publishing events for the Data Warehouse to the PubSub topic in GCP, we follow this process:

  1. Run Daily Models
    We run our daily models in the morning (Swedish timezone) via the dbt-structure-daily DAG in Airflow.

  2. Trigger Event Publisher
    Once the daily models complete, they trigger the execution of the credit-event-publisher DAG.

  3. Calculate Base & Delta Tables
    This DAG runs dbt models tagged with CreditRiskDecision:

    • Calculates the required "Base" tables
    • Creates a DELTA table comparing current fields vs. last run
    • Delta contains only fields meant for event payloads

  4. Publish Events
    The DAG publishes one event to PubSub for each row in the DELTA table.

  5. Update Archive
    Finally, the DAG updates an "archive" table holding the current state of the Base table. This serves as the benchmark for the next comparison run.

Flowchart

graph TB
    Credit[Credit Event<br/>Publisher DAG]

    Credit --> Base[(update<br/>Base Table)]

    Base --> D1[(payload Delta<br/>event 1)]
    Base --> D2[(payload Delta<br/>event 2)]

    D1 --> Get[Get payload<br/>data<br/>from BQ<br/>per event]
    D2 --> Get

    Get --> Create[Create Event<br/>Messages]
    Create --> Validate[Validate<br/>All Messages]

    Validate -->Publish[Publish<br/>to Pub/Sub]

    Publish --> Update[Update<br/>Archive]

    Update --> Success[✅ Complete]
Hold "Alt" / "Option" to enable pan & zoom