Skip to content

dbt Development Process

1. Development

  • Always work on a personal branch, never commit to main
  • Use dbt Cloud IDE, a laptop, or both. On a laptop keep the same dbt version as dbt Cloud (see the repo README.md)

2. Commits

  • Refactor before enhancing when practical
  • A pre-commit hook runs sqlfluff on updated files. In the Cloud IDE use the Lint button. Bypass with git commit --no-verify -m "<message>"
  • Prefix the commit message with the nature of the change: Bugfix, New feature, Refactoring, or a combination

3. Pull requests and review

  1. Raise the PR in GitHub targeting main, with the Linear issue link in the description, and move the issue to In Review
  2. Check the CI checks pass. Failures on incremental models are expected for now
  3. Post the PR in #team-data-internal and get it reviewed
  4. Pull main and resolve conflicts if another branch merged
  5. Address review comments, get approval in GitHub, merge to main

4. Deployment

  1. Delete your branch after the merge
  2. Make sure the dbt job is not running in dbt Cloud
  3. If needed, bump the DBT_PROD_WH warehouse size
  4. For custom refreshes, edit and run the Prod Manual Instructions job
  5. Verify the data
  6. Scale DBT_PROD_WH back to Medium. A Snowflake task also reverts it daily at 21:00 UTC

Testing and validating

Bigger models use a 7-day lookback outside production. Two variables in dbt_project.yml change this:

  • dev_load_full: boolean, run with the full dataset. Default false
  • dev_load_days: lookback window in days. Default 7
dbt run -m <yourmodel> --full-refresh --vars 'dev_load_days: 365'
dbt run --full-refresh --vars 'dev_load_full: true'

Note

Both variables are ignored in production.