Skip to content

AWS ingestion and retries

S3 file ingestion, backend events and recovery paths.

Sources / partners Airflow / dbt processing Storage / BigQuery Messaging / queues Functions / Lambda Reports / destinations Reliability / recovery

Drag to pan; use + / −, Fit or Fullscreen to inspect a flow. Dashed arrows identify failure and replay paths.

S3 file ingestion

flowchart LR
    files["S3 file feeds"] -->|"ObjectCreated"| topics["SNS topic per bucket<br/>sns-bigquery-&lt;bucket&gt;"]
    topics -->|"22 production subscriptions"| prod["Production Lambda<br/>lambda-bigquery-ingest"]
    topics -->|"18 development subscriptions"| dev["Development Lambda<br/>lambda-bigquery-ingest-dev"]
    prod --> prod_bq["Production BigQuery<br/>s3 dataset"]
    dev --> dev_bq["Development BigQuery<br/>s3 dataset"]
    class files flow-sources
    class topics flow-messaging
    class prod,dev flow-function
    class prod_bq,dev_bq flow-storage
Hold "Alt" / "Option" to enable pan & zoom

Both functions use the s3_lambda handler.

Source families

  • Remittance: gmt, italcambio, remitee, thunes, uniteller; production and development.
  • Bank and card feeds: cfsb, dwolla-reconciliation, galileo-sftp, perfect-plastic; production and development. atm-allpoints is production only.
  • Reconciliation: kard, psp-funding-reconciliation, remittance-reconciliation; production and development. rebtel is production only, imported through Airflow into Majority S3.
  • Growth and attribution: braze in production and development; adjust-app and mp-forwardingteam-app in production only.
  • Operations: azure-devops, talkdesk, telesales; production and development.
  • Data and models: fx-rates, ml-models; production and development.

Buckets follow <prefix>-majority-{prod,dev}. The production FX exception is fx-rates-majority-prod → sns-bigquery-fx-rates-majority-prod-2.

SNS message-body filters

  • Braze: currents/ and selected segment-export/ IDs.
  • CFSB: from_sftp/decrypted/.
  • Galileo: configured report prefixes.
  • Kard: kard-reconciliation/cardlinked-reconciliation/.
  • PSP funding: fiserv/{authorizations,chargebacks,fundings,settlements}/.

Backend events

flowchart LR
    stream["Kinesis<br/>ks-event-prod"] -->|"consumer-data<br/>enhanced fan-out"| handler["Backend events Lambda<br/>kinesis-lambda-bigquery-ingest"]
    handler --> warehouse["BigQuery<br/>kinesis.backend_events"]
    handler --> topic["Warehouse Pub/Sub<br/>dw-prod-gwiiag / events-topic"]
    class stream,topic flow-messaging
    class handler flow-function
    class warehouse flow-storage
Hold "Alt" / "Option" to enable pan & zoom

The handler is backend_events_lambda. Development and stage streams (ks-event-dev, ks-event-stage) use kinesis-lambda-bigquery-ingest-dev, development BigQuery and its events-topic.

The old mParticle stream and mappings were removed. Retained functions and archived buckets do not form an active ingestion path; see the historical route.

S3 failure capture and replay

flowchart LR
    failed["S3 ingestion Lambda<br/>lambda-bigquery-ingest"] -.->|"Async invocation failure"| topic["SNS dead-letter topic<br/>dlq-sns-lambda-s3"]
    topic -.->|"Raw SNS message"| queue["SQS dead-letter queue<br/>dlq-sqs-lambda-s3"]
    topic --> email["Slack email subscriber"]
    queue -.->|"Poll"| pipe["EventBridge Pipe<br/>lambda-s3"]
    pipe -.->|"Replay"| replay["Reinvoke same Lambda<br/>lambda-bigquery-ingest"]
    class failed,replay flow-function
    class topic,queue flow-messaging
    class pipe flow-reliability
    class email flow-destination
Hold "Alt" / "Option" to enable pan & zoom

The pipe is provisioned stopped and enabled by an operator, with batch size 1. Its input wrapper is {"dlq": <$.body>}. Reinvocation resumes the ingestion handler.

Backend failure capture and replay

flowchart LR
    failed["Backend events Lambda<br/>kinesis-lambda-bigquery-ingest"] -.->|"Async invocation failure"| topic["SNS dead-letter topic<br/>dlq-sns-lambda-backend-events"]
    topic -.->|"Raw SNS message"| queue["SQS dead-letter queue<br/>dlq-sqs-lambda-backend-events"]
    topic --> email["Slack email subscriber"]
    queue -.->|"Poll"| pipe["EventBridge Pipe<br/>lambda-backend-events"]
    pipe -.->|"Replay"| replay["Reinvoke same Lambda<br/>kinesis-lambda-bigquery-ingest"]
    class failed,replay flow-function
    class topic,queue flow-messaging
    class pipe flow-reliability
    class email flow-destination
Hold "Alt" / "Option" to enable pan & zoom

This pipe is also provisioned stopped, with batch size 1 and input wrapper {"dlq": <$.data>}.

Async failures versus stream retries

This dead-letter path applies to asynchronous Lambda invocations. Kinesis record retries stay on the event-source mapping and do not automatically enter this queue. No Kinesis OnFailure destination is configured for this path.

Retry queue alerts

flowchart LR
    alarms["CloudWatch DLQ alarms<br/>Visible + in-flight messages"] --> topic["SNS alert topic<br/>lambda-errors"]
    topic --> chatbot["AWS Chatbot SNS integration<br/>HTTPS subscription"]
    chatbot --> slack["Slack notifications"]
    class alarms flow-reliability
    class topic flow-messaging
    class chatbot flow-processing
    class slack flow-destination
Hold "Alt" / "Option" to enable pan & zoom

Implementation repositories: dt-aws-infrastructure and dt-bigquery-lambda. Return to data flows.