5 Projects to Land an
Infra/DevOps/Platform Role

Goal: A commit to the receipt-api source repo becomes a healthy pod on EKS without a single kubectl command from a human. Time: 15–20 hours | Cost: EKS control plane + 2 t4g.small nodes ≈ $0.11/hr

GitHub Actions → ECR → GitOps repository → Argo CD → Helm → EKS
                                                     ↑
Route 53 → ALB → healthy pods → S3                 Terraform builds the platform
                         ↓
                     CloudWatch

Phase 1.1 — Draw before building

Notes to re-read

• AWS §0 (entire chapter: “What AWS actually is”)
• AWS §2 (VPC: “Where may packets travel?”)
• TF §Expedition One (“The three-world map”)

What to build

On paper (or Mermaid in a scratch file), draw these two diagrams:

Diagram A — the runtime data path:

flowchart LR
    U["User"] -->|"DNS resolution
Route 53, UDP 53"| ALB["ALB
internet-facing"]
    ALB -->|"HTTPS :443
TLS terminated at ALB"| POD["Pod
receipt-api"]
    POD -->|"Pod Identity
IRSA, STS"| S3["S3 Bucket
receipts"]
    

Diagram B — the change delivery path:

flowchart LR
    GIT["git push
to receipt-api repo"] -->|"OIDC: GitHub Actions
IAM role"| CI["CI
Build & Push"]
    CI -->|"docker push
digest"| ECR["ECR
immutable image"]
    ECR -->|"PR updates digest
in values file"| PR["GitOps PR
receipt-deploy repo"]
    PR -->|"Argo CD watches Git
reconciliation loop"| ARGO["Argo CD
Application"]
    ARGO -->|"Helm template
then kubectl apply"| EKS["EKS
receipt namespace"]
    

Phase 1.2 — Receipt API (containerised)

Notes to re-read

• Docker §1 (First Principles — image vs container, build time vs runtime)
• Docker §4 (Images and Layers — tags vs digests)
• Docker §5 (Dockerfiles and BuildKit — multi-stage, non-root, cache order)
• Docker §10 (Security — run as non-root, read-only rootfs, drop capabilities)

What to build

receipt-api/Dockerfile — a production-minded image:

1. Multi-stage if your language benefits from it (Go, Rust: yes; Python, Node: less so).
2. Final stage runs as non-root user (UID 10001).
3. HEALTHCHECK probes /health/ready.
4. Exec-form CMD.
5. No secrets baked into layers.
6. Uses a trusted base image (not :latest floating reference).

receipt-api/app.py (or equivalent) — the API itself:

• POST /receipts → store in S3.
• GET /health/ready → return 200.
• Structured JSON logging to stdout.
• SIGTERM handler for graceful shutdown (Docker §6).
• Validates that required env vars exist at startup.

Decision checkpoint

Phase 1.3 — S3 bucket and IAM roles (Terraform Foundation)

Notes to re-read

• TF §Opening Puzzle (the rename that deletes a server)
• TF §Expedition One (init → fmt → validate → plan → apply → destroy cycle)
• TF §The language workshop (variables, locals, outputs, resource blocks)
• TF §State (the four sentences on what state actually is)
• AWS §1 (IAM: roles, trust policies, permissions policies, Pod Identity)
• AWS §4 (S3: object semantics, versioning, block public access, encryption)

What to build

A standalone Terraform root terraform/foundation/ that creates:

Decision checkpoint

Phase 1.4 — VPC, EKS, and Platform IAM

Notes to re-read

• AWS §2 (VPC — the whole chapter, especially the packet checklist)
• AWS §9 (EKS — control plane vs data plane, the three identity doors, VPC CNI)
• TF §Modules (boundaries of meaning, flat composition, moved blocks)
• K8s §3 (Cluster Architecture — control plane components, node components)
• K8s §1 (First Principles — reconciliation loops, spec vs status)

What to build

terraform/platform/ — a Terraform root that creates the cluster:

Architecture decisions to make and justify

The Terraform ownership boundary — think before you commit

Terraform should own: VPC, subnets, route tables, endpoints, EKS cluster and node groups, IAM roles and Pod Identity associations, ECR repositories, S3 buckets, CloudWatch log groups, the bootstrap Argo CD installation.

Terraform should NOT own: Helm releases inside the cluster (Argo CD owns those), Application Deployments/Services/Ingresses (Argo CD owns those), Namespaces (Argo CD with CreateNamespace=true).

Phase 1.5 — ECR, CI, and immutable image promotion

Notes to re-read

• AWS §5 (ECR: registry, repository, image, tag vs digest, tag immutability)
• AWS §11 (CI/CD: OIDC, protected environments, build once, promote the digest)
• Docker §3 (Architecture — BuildKit, registry, distribution)

What to build

terraform/ecr.tf (add to the platform root):

resource "aws_ecr_repository" "receipt_api" {
  name                 = "receipt-api"
  image_tag_mutability = "IMMUTABLE"
  force_delete         = false

  image_scanning_configuration {
    scan_on_push = true
  }
}

terraform/github-oidc.tf — OIDC provider and publisher role with trust policy scoped to your repo and environment. Permissions: ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:CompleteLayerUpload, ecr:UploadLayerPart, ecr:InitiateLayerUpload, ecr:PutImage.

.github/workflows/build.yaml in receipt-api repository:

name: Build receipt image
on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  image:
    runs-on: ubuntu-latest
    environment: production-build
    steps:
      - checkout
      - test (your language's test runner)
      - aws-credentials (OIDC → assume publisher role)
      - ecr-login
      - docker build + push
        tags:
          - git-${{ github.sha }}
        push: true
      - print digest for promotion PR

Do NOT run kubectl apply from CI. CI publishes an artifact and proposes a change. It does NOT touch the cluster.

Decision checkpoint

Phase 1.6 — Helm chart, GitOps repo, and Argo CD

Notes to re-read

• K8s §4 (Pods — lifecycle, probes, pod as atom)
• K8s §5 (Deployments — ReplicaSet relationship, rolling update parameters)
• K8s §6 (Networking — the four problems, Service types, Ingress)
• AWS §11 (Helm, GitOps repository, Argo CD, promotion is a Git diff)
• K8s §1 (reconciliation — Argo CD is a continuous reconciler of Git → cluster)

What to build

Chart structure in receipt-deploy/:

receipt-deploy/
├── charts/
│   └── receipt-api/
│       ├── Chart.yaml
│       ├── values.yaml              # defaults
│       ├── values.schema.json        # validate values shape
│       └── templates/
│           ├── _helpers.tpl
│           ├── deployment.yaml
│           ├── service.yaml
│           ├── ingress.yaml
│           ├── serviceaccount.yaml
│           └── pdb.yaml
├── environments/
│   └── prod/
│       └── receipt-api.yaml         # prod-specific overrides
└── argocd/
    └── receipt-prod.yaml            # Argo CD Application

values.yaml exposes this contract:

replicaCount: 2

image:
  repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api
  digest: sha256:...  # ← this is the truth

service:
  port: 80
  targetPort: 8080

ingress:
  host: api.example.com
  certificateArn: "arn:aws:acm:..."

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi

podSecurityContext:
  fsGroup: 10001

serviceAccount:
  name: receipt-api

deployment.yaml template key requirements:

1. Image: {{ .Values.image.repository }}@{{ .Values.image.digest }} — NOT a tag.
2. readinessProbe on /health/ready.
3. livenessProbe — lightweight, does NOT call S3 (liveness failure kills containers; an S3 outage should not kill all pods).
4. securityContext.runAsNonRoot: true, readOnlyRootFilesystem: true, capabilities.drop: [ALL].

ingress.yaml template annotations:

annotations:
  alb.ingress.kubernetes.io/scheme: internet-facing
  alb.ingress.kubernetes.io/target-type: ip
  alb.ingress.kubernetes.io/healthcheck-path: /health/ready
  alb.ingress.kubernetes.io/certificate-arn: "{{ .Values.ingress.certificateArn }}"
  alb.ingress.kubernetes.io/group.name: "{{ .Values.ingress.groupName }}"

argocd/receipt-prod.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: receipt-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_USER/receipt-deploy.git
    targetRevision: main
    path: charts/receipt-api
    helm:
      valueFiles:
        - ../../environments/prod/receipt-api.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: receipt
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

From this point on, kubectl apply is forbidden for application resources. Git is the desired state. Argo CD is the reconciler.

Decision checkpoint

Phase 1.7 — DNS, TLS, and end-to-end verification

Notes to re-read

• AWS §7 (Route 53 + ALB — the whole chapter, especially "Trace it")
• K8s §6 (Internet ↔ Service, the Ingress mental model)
• AWS §6 (CloudWatch — logs, metrics, evidence chain)

What to build

Route 53 hosted zone: A ALIAS record: api.example.com → ALB DNS name

ACM certificate: DNS-validated in the same region as the cluster. ARN passed as ingress.certificateArn in Helm values.

ALB (created by AWS Load Balancer Controller from the Ingress): Internet-facing, TLS terminated at ALB, backend HTTP. Health check path: /health/ready.

Verify end-to-end:

dig api.example.com

curl https://api.example.com/health/ready
# → {"status": "healthy"}

curl -X POST https://api.example.com/receipts \
  -H "Content-Type: application/pdf" \
  --data-binary @test-receipt.pdf
# → {"id": "abc123"}

aws s3 ls s3://receipt-prod-data/receipts/abc123.pdf

aws logs tail /receipt/prod/api --follow

The complete mental movie checklist

Phase 1.8 — Promotion workflow (CI opens a PR)

Notes to re-read

• AWS §11 (Promotion is a Git diff, the complete deployment mental movie)
• K8s §1 (spec vs status — Argo CD closes this gap)

What to build

Second GitHub Actions workflow in receipt-api repo:

name: Propose deployment
on:
  workflow_run:
    workflows: [Build receipt image]
    types: [completed]

jobs:
  propose:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - determine the digest from the build run
      - checkout receipt-deploy repo
      - update digest in environments/prod/receipt-api.yaml
      - open a pull request in receipt-deploy repo

The PR body should contain: source commit SHA and link, image digest (the immutable reference), ECR scan results, rendered manifest diff (helm template output difference), rollback digest (the previous value).

Merge strategy: Not automatic. A human reviews the PR and merges it. Merging the PR changes the desired state in Git. Argo CD sees the Git change and reconciles the cluster. The deployment is a Git merge, not a CI action.

Phase 1.9 — Rollback (by Git revert)

No code — a practiced procedure:

1. Identify the bad deploy: which GitOps commit? Which digest?
2. Revert the commit in receipt-deploy repo (or open a PR restoring the old digest).
3. Merge the revert. Argo CD renders the old digest.
4. Deployment controller performs a rolling update back to the old ReplicaSet.
5. Verify: curl https://api.example.com/health/ready, check CloudWatch error rate.

Phase 1.10 — Break it, one gate at a time

Introduce one failure, predict the symptom, find the evidence, repair it, then move to the next. Keep notes on what you observed.

Completion criterion for Project 1

"A developer pushes to receipt-api main. A CI build creates an immutable image and opens a PR in receipt-deploy with the new digest. A reviewer merges the PR. Argo CD renders the Helm chart and applies it to EKS. The Deployment controller creates a new ReplicaSet. New pods start, pass readiness, and join the ALB target group. Old pods drain. CloudWatch shows the new digest serving healthy traffic. Rollback is a Git revert of the digest change."

Stop here and verify that sentence before moving to Project 2. If any word in it is fuzzy, re-read the corresponding note section.

Goal: The Receipt API infrastructure becomes reusable. A new team can deploy a different app on the same platform by writing a single YAML spec, without knowing Terraform or EKS internals. Time: 10–15 hours | Builds on: Project 1 infrastructure

platform.yaml
  → Terraform module call (S3, ECR, IAM, CloudWatch)
  → CI workflow template (build + push)
  → Helm value generation (from the spec)
  → Argo CD Application registration

Phase 2.1 — Extract reusable Terraform modules

Notes to re-read

• TF §Modules (good modules raise the level of conversation, module contract, flat composition, moved blocks)
• AWS §1 (least privilege — each app gets its own IAM role)
• AWS §4 (bucket structure — each app gets its own S3 prefix)

What to build

modules/app-infra/ — the "golden path" module:

Resources created per app: ECR repository (immutable, scan on push), S3 bucket, IAM role + policy for pod-to-S3 access, Pod Identity association, CloudWatch log group.

Your terraform/platform/ root now looks like:

module "platform" {
  source = "./modules/eks-platform"
  # ... VPC CIDR, AZs, node types ...
}

module "receipt_api" {
  source = "./modules/app-infra"

  app_name           = "receipt-api"
  s3_write_prefix    = "receipts/"
  eks_cluster_name   = module.platform.cluster_name
  namespace          = "receipt"
  service_account    = "receipt-api"
  log_retention_days = 30
}

module "invoice_api" {
  source = "./modules/app-infra"

  app_name           = "invoice-api"
  s3_write_prefix    = "invoices/"
  eks_cluster_name   = module.platform.cluster_name
  namespace          = "invoice"
  service_account    = "invoice-api"
  log_retention_days = 30
}

No new VPC. No new EKS cluster. One module call provisions everything an app needs.

Decision checkpoint

Phase 2.2 — Developer spec ("platform.yaml")

What to build

Define a spec file that a developer fills out:

# platform.yaml
name: my-service
team: platform
environment: prod

app:
  port: 8080
  health_path: /health/ready

s3:
  write_prefix: my-data/

resources:
  cpu_request: 100m
  memory_request: 128Mi
  memory_limit: 256Mi

deploy:
  replica_count: 2
  ingress_host: my-service.example.com
  certificate_arn: "arn:aws:acm:..."

git:
  source_repo: example-org/my-service
  deploy_repo: example-org/my-service-deploy

This is NOT a Terraform variable file. It is an abstraction layer above Terraform. A script or CLI tool reads platform.yaml and generates: a Terraform module call, a Helm values file, and an Argo CD Application manifest.

Phase 2.3 — Reusable CI workflow template

What to build

A reusable GitHub Actions workflow that any app team calls:

name: Build and propose deployment

on:
  workflow_call:
    inputs:
      app_name:
        required: true
        type: string
      ecr_repository:
        required: true
        type: string
      aws_region:
        required: true
        type: string
      deploy_repo:
        required: true
        type: string
      values_path:
        required: true
        type: string
    secrets:
      role_to_assume:
        required: true

jobs:
  build-and-propose:
    # ... same logic as Project 1.8, but parameterised

An individual app repo\'s workflow becomes:

name: CI
on:
  push:
    branches: [main]

jobs:
  use-shared:
    uses: example-org/platform/.github/workflows/_build-and-propose.yaml@main
    with:
      app_name: receipt-api
      ecr_repository: receipt-api
      aws_region: eu-west-2
      deploy_repo: example-org/receipt-deploy
      values_path: environments/prod/receipt-api.yaml
    secrets:
      role_to_assume: arn:aws:iam::123456789012:role/receipt-github-ecr-publisher

Decision checkpoint

Phase 2.4 — Policy-as-code

What to build

Add validation gates to the Terraform module:

variable "s3_write_prefix" {
  type = string
  validation {
    condition     = can(regex("^[a-zA-Z0-9._/-]+/$", var.s3_write_prefix))
    error_message = "S3 write prefix must end with '/' and contain only safe characters."
  }
}

resource "aws_iam_policy" "receipt_s3" {
  # ...

  lifecycle {
    precondition {
      condition     = endswith(var.s3_write_prefix, "/")
      error_message = "S3 write prefix must end with '/' to prevent accidental broad access."
    }
  }
}

Use a policy scanning tool (Checkov, tfsec, or Open Policy Agent):

checkov -d modules/app-infra

Goal: When a receipt upload fails, you know before the user opens a ticket. You have logs, metrics, dashboards, and SLO alarms proving it. Time: 10–15 hours | Builds on: Project 1 cluster

Phase 3.1 — Structured application logs

Notes to re-read

• AWS §6 (entire chapter, especially "Logs should be structured evidence")
• Docker §6 (Logs: stdout and stderr are the contract)

What to build

Upgrade the Receipt API to emit structured JSON logs:

{
  "timestamp": "2026-07-24T10:02:11Z",
  "level": "info",
  "service": "receipt-api",
  "version": "sha256:abc...",
  "request_id": "req-7f1",
  "route": "POST /receipts",
  "duration_ms": 843,
  "status": 201,
  "object_key": "receipts/abc123.pdf",
  "s3_duration_ms": 780
}

Fields NEVER to log: AWS credentials or session tokens, full presigned URLs, receipt content bytes, secrets of any kind.

Use CloudWatch Logs Insights queries:

fields @timestamp, request_id, duration_ms, status
| filter route = "POST /receipts"
| sort @timestamp desc
| limit 20

Phase 3.2 — Prometheus and Grafana stack

What to build

Deploy via Helm or the Prometheus Operator (install via Argo CD as platform add-ons — not via Terraform):

Instrument the Receipt API with a /metrics endpoint:

• receipt_requests_total{status="201"} (counter)
• receipt_request_duration_seconds (histogram)
• receipt_s3_write_duration_seconds (histogram)
• receipt_requests_in_flight (gauge)

Add a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: receipt-api
  namespace: receipt
spec:
  selector:
    matchLabels:
      app: receipt-api
  endpoints:
    - port: http
      path: /metrics

Phase 3.3 — SLO dashboards

Notes to re-read

• AWS §6 (golden signals, SLO, error budget)
• K8s §5 (readiness probe gates traffic)

What to build

Dashboard 1 — Golden Signals (Grafana):

Define an SLO:

SLI:  successful receipt uploads / total valid receipt requests
SLO:  99.5% over a 28-day rolling window
Error budget: 0.5% of requests may fail

Decision checkpoint

Phase 3.4 — Meaningful alarms

User-impact alarms (page a person):

Diagnostic alarms (create a ticket):

Phase 3.5 — Chaos engineering

What to build

Install LitmusChaos or Chaos Mesh on the cluster. Create these experiments:

For each experiment: write the hypothesis before running it, observe the result, check if your alarms fired, note which alarms fired too late/early/not at all, update alarms accordingly.

Phase 3.6 — Runbook

Write a runbook — a short, actionable document that states what to check in a specific order. It is NOT a wiki. It is what gets you from "alarm fired" to "root cause identified" or "service restored."

Incident 1 — "Users report 503 errors"

1. What already worked? (DNS? TLS? ALB reachable?)
2. Check ALB target health: are targets healthy?
3. Check pod status: kubectl get pods -n receipt
4. Check recent events: kubectl describe deployment receipt-api -n receipt
5. Check CloudWatch / Grafana: what changed in the last 15 minutes?

Incident 2 — "Receipt uploads fail with 500"

1. What changed recently? (Check Argo CD sync status, GitOps commits)
2. Check app logs: filter for ERROR level
3. Verify Pod Identity: does the ServiceAccount still have the IAM association?
4. Check S3 bucket: does the bucket exist? Is the policy intact?
5. Check VPC endpoints: can the pod reach the S3 endpoint?
6. Check CloudTrail: who changed what? When?

Incident 3 — "Pods stuck Pending after scale-up"

1. Describe a pending pod: kubectl describe pod <name> — read the Events section
2. Is it a node capacity issue? (Insufficient CPU/memory → add nodes)
3. Is it a subnet IP issue? (VPC CNI exhausted subnet IPs)
4. Is it a taint/affinity mismatch?
5. Check node autoscaler logs if configured

Start with the incident diagnosis flowcharts in your notes. The flowcharts are architecture-neutral; your runbook adds the specific names, namespaces, and resource IDs of your cluster.

Goal: The same Receipt API image runs in three environments — Docker Compose (dev), ECS Fargate (stage), and EKS (prod) — from one CI pipeline. Time: 8–12 hours | Builds on: Project 1 CI + Project 2 modules

Phase 4.1 — Docker Compose development environment

Notes to re-read

• Docker §9 (Compose — the whole chapter, especially the everyday command loop)
• Docker §11 (the complete lifecycle trace)

What to build

compose.yaml in the receipt-api repository root:

name: receipt-dev

services:
  app:
    build:
      context: .
      target: development
    init: true
    environment:
      APP_MODE: development
      S3_ENDPOINT: http://localstack:4566
      S3_BUCKET: receipt-dev-data
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ./app.py:/app/app.py:ro     # live code reload
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health/ready"]
      interval: 5s
    depends_on:
      localstack:
        condition: service_healthy

  localstack:
    image: localstack/localstack:latest
    environment:
      SERVICES: s3
    ports:
      - "127.0.0.1:4566:4566"
    volumes:
      - ./localstack-init.sh:/etc/localstack/init/ready.d/init.sh:ro
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
      interval: 5s

Dev loop:

docker compose up -d        # start everything
docker compose logs -f app   # watch logs
# edit app.py — the bind mount syncs it, app auto-reloads
curl http://127.0.0.1:8080/health/ready
docker compose down

Phase 4.2 — ECS Fargate deployment

Notes to re-read

• AWS §8 (ECS — the whole chapter, especially task execution role vs task role, awsvpc networking, task definition revisioning)

What to build

A new Terraform root terraform/ecs-stage/ that creates:

Key learning differences from EKS

Decision checkpoint

Phase 4.3 — Multi-arch image builds

What to build

Upgrade the CI build workflow to produce a multi-arch image index:

- name: Build and push multi-platform
  uses: docker/build-push-action@v7
  with:
    context: .
    push: true
    platforms: linux/amd64,linux/arm64
    tags: |
      ${{ steps.ecr.outputs.registry }}/receipt-api:git-${{ github.sha }}

ECR stores a manifest list pointing to platform-specific manifests. When a node pulls, an x86 node gets linux/amd64; an ARM node (t4g) gets linux/arm64.

docker buildx imagetools inspect \
  123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api:git-abc123

Phase 4.4 — Tiered CI pipeline

What to build

A single CI pipeline with environment gates:

jobs:
  test:
    # Runs on every push to any branch
    # No AWS credentials needed

  build-dev:
    needs: test
    if: github.ref == 'refs/heads/main'
    # Build with tag git-<sha>-dev
    # Uses development ECR repository

  build-prod:
    needs: build-dev
    environment: production-build
    # Build with tag git-<sha>
    # Uses production ECR repository
    # Opens the promotion PR in receipt-deploy

Key principles

1. PR code is untrusted input (AWS §11). Never give a PR branch workflow production AWS credentials.
2. Build once, promote the same digest (Docker §1). The dev and prod builds should produce identical digests. Only the tag differs.
3. Environment gates (AWS §11). The production-build environment requires approved reviewers before the job runs.

Goal: An operator that watches pod failures, captures diagnostic evidence, and takes bounded recovery actions — without human intervention. Time: 12–18 hours | Learning curve: This is the hardest project. It demands K8s internals, Go programming, and the reconciliation-loop mindset from K8s §1.

Phase 5.1 — Design the operator\'s contract

Notes to re-read

• K8s §1 (reconciliation loops — the thermostat analogy, level-based not edge-triggered)
• K8s §3 (controller-manager — the loop runner)
• K8s §5 (self-healing trace — what a controller does)

What to design

An operator is a custom Kubernetes controller. It watches resources, observes their state, and acts to close the gap between actual and desired.

The CRD you\'ll create — IncidentResponder:

apiVersion: remediation.example.com/v1
kind: IncidentResponder
metadata:
  name: crash-loop-responder
  namespace: receipt
spec:
  watch:
    namespace: receipt
    podLabelSelector:
      app: receipt-api
  triggers:
    - type: CrashLoopBackOff
      maxRestarts: 3
      action: captureAndRestart
    - type: OOMKilled
      action: captureOnly
    - type: ImagePullBackOff
      action: notifyOnly
  evidence:
    capturePodLogs: true
    captureNodeConditions: true
    captureRecentEvents: true
  actions:
    restartThreshold: 3
    cordonNodeOnRepeatedFailure: true
    maxCordonNodes: 1
  notifications:
    slackWebhookUrlRef:
      name: slack-webhook
      key: url

Spec explanation:

Status (written by the operator, read by you):

status:
  observedGeneration: 1
  conditions:
    - type: Ready
      status: "True"
      lastTransitionTime: "2026-07-24T10:00:00Z"
  lastIncidents:
    - timestamp: "2026-07-24T09:55:00Z"
      pod: receipt-api-7d8f9-abc12
      trigger: OOMKilled
      action: captureOnly
      summary: "Pod OOMKilled. Logs captured. Memory limit was 128Mi; pod was using 300Mi+."

Phase 5.2 — Scaffold the Go project with controller-gen

What to build

Use kubebuilder or operator-sdk to scaffold:

mkdir incident-operator && cd incident-operator
kubebuilder init --domain example.com --repo github.com/YOUR_USER/incident-operator
kubebuilder create api \\
  --group remediation --version v1 --kind IncidentResponder \\
  --resource --controller

This generates:

incident-operator/
├── api/v1/
│   ├── incidentresponder_types.go   # ← you fill in Spec/Status structs
│   ├── groupversion_info.go
│   └── zz_generated.deepcopy.go
├── internal/controller/
│   └── incidentresponder_controller.go  # ← you fill in Reconcile
├── config/
│   ├── crd/          # CRD YAML
│   ├── rbac/         # RBAC for the operator
│   └── manager/      # Deployment for the operator itself
├── main.go
├── go.mod
├── Makefile
└── Dockerfile

Let the LLM agent generate the boilerplate. Your job is filling in the Reconcile function and the notification/action logic.

Phase 5.3 — Implement the reconciliation loop

Notes to re-read

• K8s §1 (the controller pattern: observe, compare, act, repeat)
• K8s §5 (the self-healing trace)
• K8s §9 (cordon, eviction, QoS)

What the Reconcile function does

Reconcile(ctx, req) → (ctrl.Result, error):

1. Fetch the IncidentResponder CR for this request
2. List pods matching the watch selector
3. For each pod:
   a. Check container statuses for CrashLoopBackOff, OOMKilled, ImagePullBackOff
   b. If a trigger matches and the threshold is exceeded:
      i.   Capture evidence: pod logs, node conditions, recent events
      ii.  Format an incident summary
      iii. Take the configured action:
           - captureOnly: log the summary, update CR status
           - captureAndRestart: capture, then delete the pod (Deployment recreates it)
           - notifyOnly: capture, post to Slack/PagerDuty
      iv.  If cordonNodeOnRepeatedFailure and same-node failures exceed threshold:
           Cordon the node
4. Update CR status with incident history
5. Requeue after a configurable interval

Go code structure

func (r *IncidentResponderReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cr remediationv1.IncidentResponder
    if err := r.Get(ctx, req.NamespacedName, &cr); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    var pods corev1.PodList
    selector, _ := metav1.LabelSelectorAsSelector(&cr.Spec.Watch.PodLabelSelector)
    r.List(ctx, &pods, client.InNamespace(cr.Spec.Watch.Namespace),
        client.MatchingLabelsSelector{Selector: selector})

    for _, pod := range pods.Items {
        for _, containerStatus := range pod.Status.ContainerStatuses {
            if match := evaluateTriggers(cr.Spec.Triggers, containerStatus, pod); match != nil {
                evidence := captureEvidence(ctx, r, pod, containerStatus)
                takeAction(ctx, r, match.Action, pod, evidence)
                sendNotification(cr.Spec.Notifications, evidence)
            }
        }
    }

    r.Status().Update(ctx, &cr)
    return ctrl.Result{RequeueAfter: cr.Spec.RequeueInterval.Duration}, nil
}

What you must understand (not the LLM)

1. Level-based, not edge-triggered: If the operator crashes after capturing evidence but before updating status, the next reconciliation pass sees the same pod state and re-evaluates. (K8s §1)
2. Idempotent actions: Calling deletePod twice should be safe. The second delete on an already-gone pod should not fail the reconciliation.
3. Status is the output: The CR\'s status.lastIncidents is how you know what the operator did. Never hide incidents in local logs.

Phase 5.4 — Evidence capture

What to build

type IncidentEvidence struct {
    Timestamp           metav1.Time
    PodName             string
    PodNamespace        string
    NodeName            string
    Trigger             string
    ContainerName       string
    RestartCount        int32
    ImageDigest         string

    PodLogsTail         string   // last N lines
    NodeConditions      []string // relevant node conditions
    RecentEvents        []string // events for this pod
    PreviousContainerLog string  // logs from --previous if CrashLoopBackOff
}

Capture functions: Pod logs (tail, last 50 lines), Node conditions (True status ones), Recent events for this pod, Previous container logs (if CrashLoopBackOff).

Phase 5.5 — Actions and notifications

What to build

Action implementations:

type ActionType string

const (
    ActionCaptureOnly      ActionType = "captureOnly"
    ActionCaptureAndRestart ActionType = "captureAndRestart"
    ActionNotifyOnly      ActionType = "notifyOnly"
)

func takeAction(ctx context.Context, r client.Client, action ActionType,
                 pod corev1.Pod, evidence IncidentEvidence) error {
    switch action {
    case ActionCaptureOnly:
        return nil  // Evidence appears in CR status

    case ActionCaptureAndRestart:
        return r.Delete(ctx, &pod)
        // The owning controller (Deployment/ReplicaSet) recreates it

    case ActionNotifyOnly:
        return nil  // Evidence sent via notification channel
    }
    return nil
}

Cordon node logic:

func maybeCordonNode(ctx context.Context, r client.Client,
                      cr Remediationv1.IncidentResponder, nodeName string) error {
    recentOnNode := countRecentIncidentsOnNode(cr, nodeName)
    if recentOnNode < cr.Spec.Actions.RestartThreshold {
        return nil
    }

    var node corev1.Node
    r.Get(ctx, client.ObjectKey{Name: nodeName}, &node)
    node.Spec.Unschedulable = true
    return r.Update(ctx, &node)
}

Notification: Post to Slack via webhook with incident summary: namespace, node, trigger, restarts, image digest, action taken, log tail.

Phase 5.6 — Helm chart for the operator

What to build

A Helm chart charts/incident-operator/ with:

incident-operator/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── serviceaccount.yaml
    ├── clusterrole.yaml
    ├── clusterrolebinding.yaml
    ├── deployment.yaml
    └── crd/
        └── incidentresponders.yaml

RBAC template — least privilege:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: incident-operator
rules:
  - apiGroups: [remediation.example.com]
    resources: [incidentresponders]
    verbs: [get, list, watch]
  - apiGroups: [remediation.example.com]
    resources: [incidentresponders/status]
    verbs: [update]

  - apiGroups: [""]
    resources: [pods, pods/log]
    verbs: [get, list, watch]
  - apiGroups: [""]
    resources: [pods]
    verbs: [delete]

  - apiGroups: [""]
    resources: [nodes]
    verbs: [get, list, watch, update]

  - apiGroups: [""]
    resources: [events]
    verbs: [get, list, watch]

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: incident-operator
  namespace: incident-operator-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: incident-operator
  template:
    spec:
      serviceAccountName: incident-operator
      containers:
        - name: operator
          image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 128Mi

Phase 5.7 — Deploy, trigger, observe

What to build — a test plan

1. Deploy the operator via Argo CD alongside Receipt API.

2. Create an IncidentResponder CR:

apiVersion: remediation.example.com/v1
kind: IncidentResponder
metadata:
  name: receipt-responder
  namespace: receipt
spec:
  watch:
    namespace: receipt
    podLabelSelector:
      app: receipt-api
  triggers:
    - type: OOMKilled
      maxRestarts: 3
      action: captureOnly
    - type: CrashLoopBackOff
      maxRestarts: 5
      action: captureAndRestart
  evidence:
    capturePodLogs: true
    captureNodeConditions: true
    captureRecentEvents: true
  actions:
    restartThreshold: 5
    cordonNodeOnRepeatedFailure: true
    maxCordonNodes: 1
  notifications:
    slackWebhookUrlRef:
      name: slack-webhook
      key: url
  requeueInterval: 30s

3. Trigger each scenario:

4. Verify:

kubectl get incidentresponder receipt-responder -n receipt -o yaml

Completion criterion for Project 5

"A pod crashes with OOMKilled. The kubelet restarts it. After three restarts, the IncidentResponder operator\'s Reconcile loop notices the container status matches the OOMKilled trigger. It captures pod logs, node conditions, and recent events. It writes an incident summary to the CR status. It does NOT delete the pod (action is captureOnly). The incident appears on the Slack channel. An on-call engineer sees it, checks the Grafana dashboard, and increases the pod\'s memory limit in the Helm values. Argo CD applies the change. The next deployment uses the correct resource limits."