Skip to content

RDF (Reporting Data Feed) WIP

Stakeholders

External Recipient: Majority (internal — RDF consumers)

Product Owner (Debit): Martina Huzell

Product Owner (Credit): [TBD]

Technical Owner: Data Engineering

Notify on failure or delay: Sara Mosleh

Purpose

Consolidated reporting data feeds providing a unified, enriched view of Majority account, card, customer, and transaction data. Used as the authoritative source for internal analytics, audit, and downstream integrations.

Finance uses these files daily to perform money movements, and the Axiom and WebBank bank reports are built on top of them. A missed or late delivery blocks both.

Data

  • majority_account_and_card — One row per active or blocked account-card pair. Consolidated account and card status with identifiers, PRN, account type, ledger type, and timestamps.
  • majority_authorized_transaction — One row per authorized (non-settled) ledger transaction. Authorization-stage transactions with amounts, balances, and metadata.
  • majority_customer — One row per KYC-verified active user. Master customer record including personal details, address, contact info, and key timestamps in Mountain Time. Excludes canceled accounts.
  • majority_settled_transaction — One row per settled ledger transaction. Comprehensive settled transaction view joining processor and ledger data, including merchant details, interchange fees, and balance impact.
  • majority_settled_transaction_audit_2025 — One row per settled transaction in calendar year 2025 (Mountain Time). Audit-purpose model unifying Galileo (pre-December 2025) and Majority (December 2025 onward) transaction sources with a source field identifying origin.

Delivery

  • Frequency: Daily
  • Schedule: 0 10 * * * (UTC) — Airflow DAG majority_rdfs
  • Weekend Delivery: Yes — plain daily cron, no weekday gate. Files are produced every day including weekends and holidays.
  • Empty Files: No — if the query returns no rows, dt-reporting logs a warning, skips the upload and exits successfully. No file lands in GCS and the Airflow task still turns green, so an empty result is silent. Consumers must treat a missing file as "no data", not as a failure.
  • Destination: GCS bucket majority-reports-prod only (SFTP_UPLOAD_DISABLED=true). Default report_date is yesterday; the report_date and task_selector params allow backfilling a single report.

Downstream Dependencies

Two DAGs wait on majority_rdfs with an ExternalTaskSensor (external_task_id=None, execution_delta=3h, failed_states=["failed"], timeout 3h):

DAG Schedule Impact if majority_rdfs fails
axiom_settlement_reports_sftp_transfer 0 13 * * * Sensor fails, no Axiom settlement reports delivered to the bank SFTP. See Axiom.
webbank_sftp_upload_reports 0 13 * * * Sensor fails, no WebBank reports (incl. the Tuesday waterfall report) delivered. See WebBank.

Because failed_states includes failed, a failed majority_rdfs run fails the sensors immediately rather than letting them wait — both bank deliveries are blocked the same day.

Common Issues

Missing event in processor_callback

ProcessorCallbackSettlementReportingEvent records never arrive from the source for one or more settlements. processor_callback_settlement_reporting_event (and therefore fact_processor_settlement and majority_settled_transaction) is missing those transactions, so the RDF files under-report settled volume.

Repository

GitHub Repository: dt-reporting (report generation and upload), dt-airflow-dags (orchestration), dt-dbt (models and tests)

Change Log

Date Change Author
2026-07-30 Added owners and notification contact, downstream DAG dependencies, common issues, weekend and empty-file behaviour Data Engineering
2026-03-02 Initial documentation Data Engineering