Kubernetes

We use Kubernetes to run our Apache Airflow instances in Azure AKS (Azure Kubernetes Service).

Overview

Clusters:

Namespace: datateam

Primary Use: Running Airflow scheduler, webserver, and worker pods

Install the Kubernetes command-line tool:

brew install kubectl

Verify installation:

kubectl version --client

See the Azure setup guide for Azure CLI installation and authentication.

Ensure you're connected to Warp VPN before accessing Kubernetes resources.

Install the Kubernetes extension for VS Code.

This extension allows you to:

Connect to the Stage cluster:

az aks get-credentials --resource-group stage-rg --name stage-bankV2-use2-aks

Connect to the Production cluster:

az aks get-credentials --resource-group prod-rg --name prod-bankV2-use2-aks

List available contexts:

kubectl config get-contexts

You should see both clusters listed.

Set the current cluster context:

# Switch to stage
kubectl config use-context stage-bankV2-use2-aks

# Switch to prod
kubectl config use-context prod-bankV2-use2-aks

Set the Data Team namespace as default:

kubectl config set-context --current --namespace=datateam

Now all commands will use the datateam namespace by default.

View all pods in the current namespace:

kubectl get pods

View pods in a specific namespace:

kubectl get pods -n datateam

View with more details:

kubectl get pods -o wide

Get detailed information about a pod:

kubectl describe pod pod-name

Stream logs from a pod:

kubectl logs -f pod-name

Open a shell in a pod:

kubectl exec -it pod-name -- /bin/bash

Run a single command:

kubectl exec pod-name -- ls -la /app

Forward a local port to a pod:

kubectl port-forward pod-name 8080:8080

Now you can access the pod's service at localhost:8080.