Amazon Web Services (AWS)

We use AWS for streaming events from backend systems and exporting data to third-party systems like Braze.

Overview

Account: Majority BI
Access Method: SAML SSO via Google
Role: AdministratorAccess
Primary Services: S3, Kinesis, Lambda

Web Console Access

Accessing AWS via Google Apps

Open Chrome and click your profile icon (or Google Apps menu)
Scroll down and locate "Amazon Web Services"
Click it to access AWS
You'll see the Majority BI account listed
Select the account with AdministratorAccess role

Bookmark It

Bookmark the AWS console for quick access. The login via Google Apps is the recommended method.

CLI Setup

Install AWS CLI

Install the AWS CLI using Homebrew:

brew install awscli

Verify the installation:

aws --version

Configure SSO

Configure the AWS CLI with SSO credentials:

aws configure sso

Provide these values when prompted:

sso_session = MajorityBI
sso_start_url = https://d-9067d9f1c8.awsapps.com/start/#
sso_region = us-east-1
sso_registration_scopes = sso:account:access
sso_role_name = AdministratorAccess

The CLI will open your browser for authentication. Sign in with your Majority account.

Verify Configuration

After setup, your ~/.aws/config file should look like this:

[default]
sso_session = MajorityBI
sso_account_id = 391008507910
sso_role_name = AdministratorAccess
region = us-east-1
output = json

[sso-session MajorityBI]
sso_start_url = https://d-9067d9f1c8.awsapps.com/start/#
sso_region = us-east-1
sso_registration_scopes = sso:account:access

Refreshing Credentials

Session Expiration

Your SSO session expires after 8-12 hours. When you see authentication errors, refresh your credentials.

To get fresh credentials:

aws sso login

This will: 1. Open your browser 2. Prompt you to authenticate 3. Refresh your access tokens

You'll need to run this roughly once per day.

Verify Access

Test your CLI access by listing S3 buckets:

aws s3 ls

You should see a list of buckets in the Majority BI account.

Common Commands

S3 Operations

# List buckets
aws s3 ls

# List objects in a bucket
aws s3 ls s3://bucket-name/

Lambda Operations

# List functions
aws lambda list-functions

# Invoke a function
aws lambda invoke --function-name function-name output.json

Rotating AWS Keys

Every 90 days we need to rotate AWS keys manually for the following IAM users and integrations:

Adjust

IAM User: majority-adjust

Update keys in: - Adjust App 1 - Adjust App 2

Braze

IAM User: majority-braze

Update keys in: - Braze Currents Integration - Braze Technology Partners

mParticle Dev Kinesis

IAM User: majority-ks-mparticle-event-dev

Update keys in: - mParticle Dev Event Output

mParticle Prod Kinesis

IAM User: majority-ks-mparticle-event-prod

Update keys in: - mParticle Prod Event Output

mParticle Sales Tool & Quarantine

IAM User: majority-mparticle

Update keys in: - mParticle Sales Tool Output - mParticle Quarantine Output

Accessing AWS Resources from Airflow DAGs

We have set up trust cloud between AKS (where the DAGs run) and AWS. This eliminates the need for secrets in DAGs for AWS access.

Default AWS Role

For most DAGs, we use the default AWS role:

arn:aws:iam::391008507910:role/xxxx-aks-airflow-role

This role already exists and is configured for standard use cases.

Steps for Custom AWS Role Access

Follow these steps only if you need to run DAGs with a different role than the default:

1. Add OpenTofu Configurations (Conditional)

Add OpenTofu configurations to create the necessary resources. See this PR for reference.

This will create: - Kubernetes service account - AWS role

Note: You need 2 PR approvals. Work with the platform team to get this OpenTofu deployed.

2. Configure AWS Permissions

In AWS, assign the required permissions to the newly created AWS role.

3. Update Airflow DAGs

[TODO] add it in airflow documentation

In the dt-airflow-dags repository, ensure your DAGs have the proper configuration: - Volume - Volume mount - Environment variables