AWS & DevOps, A to Z

Start with one process

On your laptop:

    flowchart LR
        B["Browser / curl"] -->|"POST :8080/receipts"| APP["Receipt API process"]
        APP -->|"write file"| DISK["./receipts/abc.pdf"]

This works while:

• your laptop is on;
• the process has not crashed;
• port 8080 is reachable;
• the disk has space;
• the file is not lost;
• one process can handle the traffic.

The moment another person depends on it, you inherit questions:

AWS does not give one magic answer. It gives services that solve different boundaries.

AWS is a collection of APIs

The AWS console is a visual client for AWS APIs.

    flowchart LR
        CONSOLE["AWS Console"]
        CLI["AWS CLI"]
        TF["Terraform"]
        SDK["Application SDK"]
        API["AWS service APIs"]
    
        CONSOLE --> API
        CLI --> API
        TF --> API
        SDK --> API

Creating a bucket in the console and creating it through Terraform both eventually become API calls. This gives us one reusable debugging sentence:

Which principal called which API action
    on which resource under which conditions?

Keep that sentence. It will explain IAM, Terraform errors, CloudTrail events, ECR pushes, and the Receipt API's S3 call.

Account, Region, and Availability Zone

Think in containment:

    flowchart TB
        ORG["Organization"]
        ACCOUNT["AWS account
ownership, IAM, billing, quotas"]
        REGION["Region
geographic area"]
        AZA["Availability Zone A
failure domain"]
        AZB["Availability Zone B
failure domain"]
    
        ORG --> ACCOUNT --> REGION
        REGION --> AZA
        REGION --> AZB

Exact meanings:

• Account: a strong ownership, access, billing, and quota boundary.
• Region: a geographic area containing multiple Availability Zones.
• Availability Zone: an isolated failure domain inside a Region.

A VPC spans a Region. A subnet is created in one AZ. An EC2 instance runs in one AZ.

Mental movie

Imagine three Receipt API copies:

copy 1 → node A → AZ A
    copy 2 → node A → AZ A
    copy 3 → node A → AZ A

There are three processes, but one failed node or AZ can remove all three.

Now:

copy 1 → node A → AZ A
    copy 2 → node B → AZ B
    copy 3 → node C → AZ C

Replication became fault isolation only when placement changed.

Prediction

A VPC contains two subnets. Does that mean it uses two AZs?

If S3 is a regional service, is an S3 bucket automatically copied to another Region?

Shared responsibility

AWS operates the physical cloud. What you operate depends on the abstraction:

“Managed” means the boundary moved. It never means access, data, and application design stopped being your responsibility.

Lab 0 — draw before building

Draw only this:

user → Receipt API → receipt file

Under each arrow write:

How is the destination found?
    Can a route reach it?
    Who is making the request?
    Why is that request allowed?
    Where would success/failure be visible?

Do not continue until you see that a simple arrow hides several independent contracts.

Feel the problem first

We want four actors:

1. you may create learning infrastructure;
2. GitHub Actions may push a Receipt image to ECR;
3. an EKS node may join the cluster and pull images;
4. the Receipt pod may write only under one S3 prefix.

If all four share one administrator access key:

• a leaked CI secret can delete infrastructure;
• a compromised pod can create IAM users;
• an operator action and an application action look identical;
• rotation becomes disruptive;
• least privilege is impossible.

IAM exists to give each actor an identity and a deliberately limited capability.

Mental picture: a building with temporary job badges

The analogy is useful, but now learn the exact model.

Authentication versus authorization

    flowchart LR
        CALLER["Caller"] --> AUTHN{"Authentication
Who are you?"}
        AUTHN -->|"invalid"| NO1["No usable AWS identity"]
        AUTHN -->|"valid"| AUTHZ{"Authorization
May you do this action
on this resource?"}
        AUTHZ -->|"deny"| NO2["AccessDenied"]
        AUTHZ -->|"allow"| API["AWS API performs action"]

A valid login does not mean every action is allowed. AccessDenied often proves authentication succeeded and authorization rejected the operation.

Users, roles, sessions, and STS

• IAM user: long-lived identity in one account. Avoid as the default for humans and

automation.

• IAM role: a permission-bearing identity that an approved principal can assume.
• Role session: the temporary identity created when a role is assumed.
• AWS STS: issues temporary access key, secret key, and session token.

    sequenceDiagram
        participant H as Human or workload
        participant IDP as Identity provider
        participant STS as AWS STS
        participant AWS as AWS service
    
        H->>IDP: Authenticate
        IDP-->>H: Identity proof
        H->>STS: Assume approved role
        STS-->>H: Temporary credentials
        H->>AWS: Signed API request
        AWS-->>H: Result or AccessDenied

Temporary credentials expire. You do not manually copy them into an application configuration.

A role has two different policy questions

    flowchart LR
        GITHUB["GitHub job"] -->|"Can I become this role?"| TRUST["Trust policy"]
        TRUST -->|"yes"| SESSION["Temporary role session"]
        SESSION -->|"What may I now do?"| PERMISSIONS["Permissions policy"]
        PERMISSIONS --> ECR["ECR repository"]

1. Trust policy: who may assume the role?
2. Permissions policy: what may the resulting role session do?

This distinction is vital:

GitHub trusted, but no ECR permission
    → role assumption succeeds, ECR push is denied
    
    ECR permission exists, but GitHub not trusted
    → role assumption fails before ECR is called

Read a policy as a sentence

{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "WriteReceipts",
          "Effect": "Allow",
          "Action": "s3:PutObject",
          "Resource": "arn:aws:s3:::receipt-prod-data/receipts/*"
        }
      ]
    }

Sentence:

Allow putting an object under receipts/ in exactly one bucket.

Policy fields:

Notice the object ARN ends in /receipts/*. The bucket ARN alone does not mean every object.

How policy evaluation feels

Start with no permission:

implicit deny

Then:

    flowchart TD
        START["Implicit deny"] --> ALLOW{"Applicable allow?"}
        ALLOW -->|"no"| DENY["Denied"]
        ALLOW -->|"yes"| EXPLICIT{"Applicable explicit deny?"}
        EXPLICIT -->|"yes"| DENY
        EXPLICIT -->|"no"| OTHER{"Boundary, organization policy,
session policy, resource policy
still permit it?"}
        OTHER -->|"no"| DENY
        OTHER -->|"yes"| OK["Allowed"]

An explicit deny wins. Debugging only the role's attached policy can miss:

• S3 bucket policy;
• ECR repository policy;
• VPC endpoint policy;
• KMS key policy;
• permissions boundary;
• session policy;
• organization service control policy.

The identities in our final system

If two rows have different jobs, they should not casually share one role.

Workload credentials: never paste keys

The same application code can use the AWS SDK default credential chain:

on EC2 → instance profile supplies credentials
    on ECS → task role supplies credentials
    on EKS → Pod Identity supplies credentials

The code asks the SDK for credentials. The environment supplies temporary ones.

Bad:

AWS_ACCESS_KEY_ID copied into .env
    AWS_SECRET_ACCESS_KEY committed or stored in an image

Good:

workload identity → temporary session → automatic refresh

Trace it — a pod writes one receipt

    sequenceDiagram
        participant P as Receipt pod
        participant PI as EKS Pod Identity
        participant STS as AWS STS
        participant S3 as S3
    
        P->>PI: SDK requests credentials
        PI->>STS: Obtain role session
        STS-->>PI: Temporary credentials
        PI-->>P: Credentials through provider
        P->>S3: Signed PutObject receipts/abc.pdf
        S3->>S3: Evaluate role + bucket + endpoint + key controls
        S3-->>P: Success or AccessDenied

The pod never needs a permanent AWS secret.

Break it — three failures that look similar

Failure A

NoCredentialProviders

The SDK could not obtain a usable identity. Check service account, Pod Identity association/agent, SDK support, or the selected credential provider.

Failure B

AccessDenied on s3:PutObject

Credentials exist, but authorization failed. Identify the actual caller, action, object ARN, and every applicable policy.

Failure C

connection timeout to S3 endpoint

This may be DNS/routing/endpoint/egress, not IAM. Authorization cannot be evaluated if the request never reaches the API.

IAM debugging ladder

1. Which credential source did the SDK/CLI choose?
    2. What does aws sts get-caller-identity return?
    3. Was role assumption trust accepted?
    4. Is the exact action allowed?
    5. Does the resource ARN match?
    6. Is there an explicit deny?
    7. Does a resource/endpoint/KMS policy add another gate?
    8. What does CloudTrail show?

Prediction

The Receipt pod role can write S3, but the container image cannot be pulled from ECR. Should you add ECR access to the pod role?

A GitHub OIDC role has ecr:*, but STS rejects AssumeRoleWithWebIdentity. Which policy side is wrong?

Say it back

Without saying “IAM manages access,” complete:

IAM lets me give __________ a temporary identity that may call __________ on __________ only when __________.

One good answer:

IAM lets me give a GitHub job a temporary identity that may push image layers to one ECR repository only when the OIDC token comes from the protected repository and environment.

Lab 1 — prove identity before permissions

Use a sandbox account and an existing lab role. Do not begin by reading every attached policy. Begin by proving which identity the CLI is using:

aws configure list
    aws sts get-caller-identity

Record the returned account and ARN. Now switch to the lab role or profile and run the identity check again:

aws sts get-caller-identity --profile receipt-lab

The ARN should change to an assumed-role session. That is evidence that authentication and trust succeeded; it says nothing yet about S3 authorization.

Ask the lab owner for two prepared object locations:

Allowed: s3://<lab-bucket>/receipts/iam-lab.txt
    Denied:  s3://<lab-bucket>/private/iam-lab.txt

Write the same harmless file to both locations:

printf 'identity test\n' > /tmp/receipt-iam-lab.txt
    
    aws s3 cp /tmp/receipt-iam-lab.txt \
      s3://<lab-bucket>/receipts/iam-lab.txt \
      --profile receipt-lab
    
    aws s3 cp /tmp/receipt-iam-lab.txt \
      s3://<lab-bucket>/private/iam-lab.txt \
      --profile receipt-lab

Expected result:

If CloudTrail is available, find the two PutObject events and compare principal, resource, and error fields with your prediction.

Completion criterion: without opening the policy first, you can use identity output, API results, and CloudTrail evidence to state which IAM gate passed or failed. Then inspect the policy and confirm why.

Feel the problem first

The Receipt API will need:

• inbound requests from an ALB;
• outbound calls to S3;
• DNS resolution;
• no direct inbound internet connection to worker nodes;
• two AZs;
• controlled administration.

“Put it in a VPC” does not answer any of those. The VPC is the place where we design addresses, routes, gateways, filters, and DNS.

Mental picture: a city

Now replace the analogy with the exact packet model.

The packet checklist

Every connection has:

source IP and port
    destination IP and port
    name resolution
    outbound route
    filtering
    return route
    application listening on the destination

If any one is wrong, the connection fails.

CIDR without fear

We choose:

VPC             10.20.0.0/16
    public-a        10.20.0.0/24
    public-b        10.20.1.0/24
    private-app-a   10.20.10.0/24
    private-app-b   10.20.11.0/24

The /16 is the large address range. Each /24 is a smaller, non-overlapping slice.

For now remember:

• smaller prefix number → larger range;
• subnets must fit within the VPC CIDR;
• connected networks should not overlap;
• each subnet belongs to one AZ;
• addresses are capacity—EKS pods can consume subnet IPs.

Public and private are route properties

    flowchart TB
        INTERNET["Internet"]
        IGW["Internet gateway"]
    
        subgraph VPC["VPC 10.20.0.0/16"]
            subgraph PUBLIC["Public subnet A"]
                ALB["Internet-facing ALB"]
                NAT["NAT gateway"]
                PUBRT["Route table
10.20.0.0/16 → local
0.0.0.0/0 → IGW"]
            end
    
            subgraph PRIVATE["Private app subnet A"]
                NODE["EKS node / ECS task"]
                PRIVRT["Route table
10.20.0.0/16 → local
0.0.0.0/0 → NAT"]
            end
        end
    
        INTERNET <--> IGW
        IGW <--> ALB
        NODE --> NAT --> IGW --> INTERNET

A subnet is public when its route table has a route to an internet gateway. Its name does not decide.

For IPv4 inbound internet reachability, the resource also needs a public address and filters that allow the flow. A private IPv4 address plus an internet-gateway route is not itself publicly reachable.

Route tables: destination first

Example:

The most specific matching destination wins. 10.60.5.8 matches /16 more specifically than /0.

Each subnet uses one associated route table. Several subnets may share a route table.

Internet gateway versus NAT gateway

Internet gateway

Allows appropriately addressed and routed public resources to communicate with the internet.

Public NAT gateway

Lets private IPv4 resources initiate outbound connections. It translates the source address so responses can return. It does not accept unsolicited inbound connections to the private workload.

    sequenceDiagram
        participant P as Private pod/node
        participant N as NAT gateway
        participant I as Internet gateway
        participant R as External repository/API
    
        P->>N: Outbound connection
        N->>I: Translate private source to NAT public address
        I->>R: Send request
        R-->>I: Response
        I-->>N: Response to NAT address
        N-->>P: Translate back to private destination

For resilient multi-AZ design, use a NAT gateway per used AZ when NAT is required and route each private subnet to its same-AZ NAT. One NAT in AZ A makes AZ B's egress depend on AZ A.

NAT gateways cost money. Do not create one because every diagram on the internet has one.

VPC endpoints

The Receipt workload calls AWS APIs such as S3, ECR, STS, and CloudWatch Logs. Supported VPC endpoints can provide private paths without a public IP, internet gateway path, or NAT for that service.

    flowchart LR
        POD["Private workload"] --> DNS["Private DNS"]
        DNS --> EP["VPC endpoint"]
        EP --> AWS["AWS service API"]

Endpoint types, policies, hourly charges, AZ count, and data charges differ. The lesson is not “endpoints are always better.” It is:

A private workload needs an intentionally designed path to every dependency.

Security groups: stateful doors

    flowchart LR
        USER["Internet"] -->|"TCP 443"| ALBSG["ALB security group"]
        ALBSG -->|"TCP 8080
source = ALB SG"| APPSG["App security group"]
        APPSG --> POD["Receipt pod/task/instance"]

The application rule should say:

inbound TCP 8080
    source = ALB security group

It should not say:

inbound TCP 8080
    source = 0.0.0.0/0

Security groups are stateful. Return traffic for an allowed connection is automatically allowed.

They contain allow rules, not explicit deny rules. Rules from multiple attached groups combine.

Network ACLs: stateless subnet checkpoints

Because NACLs are stateless, return traffic needs matching rules. Incorrect ephemeral port rules create mysterious timeouts.

Start with security groups. Add custom NACL complexity only for a clear requirement.

DNS happens before most network connections

    sequenceDiagram
        participant A as Application
        participant D as Route 53 Resolver
        participant R as Route table
        participant S as S3 endpoint
    
        A->>D: Resolve s3.eu-west-2.amazonaws.com
        D-->>A: Address
        A->>R: Packet to address
        R->>S: Chosen network path

If DNS fails, changing an inbound security-group rule is unlikely to help.

Trace it — ALB to private workload

1. ALB chooses target 10.20.10.31:8080.
    2. VPC local routing knows 10.20.10.31 is inside the VPC.
    3. ALB security group permits outbound 8080.
    4. workload security group permits inbound 8080 from ALB security group.
    5. any NACL permits request and return ports.
    6. process listens on 0.0.0.0:8080.
    7. process returns a response.

The connection needs all seven.

Break it

Prediction

An instance is in a subnet named private, but it has a public IP and its route table uses 0.0.0.0/0 → internet gateway. Is it private?

An application sends HTTPS outbound through a security group. Must its inbound rules allow the response's ephemeral port?

Say it back

Complete:

A VPC does not “give internet.” It gives me __________, while I deliberately add __________ for each required flow.

Good answer:

A VPC gives me an address and routing domain, while I deliberately add subnets, routes, gateways/endpoints, DNS, and filters for each required flow.

Lab 2 — packet worksheet

For each flow, fill every column:

Completion criterion: you can explain why the last column is not the same for all four flows.

Feel the problem first

Our laptop is not a production host:

• it sleeps;
• it is behind a home/office network;
• it is manually configured;
• other operators cannot reproduce it;
• replacing it may lose files;
• it has no intentional AWS identity.

EC2 gives us a virtual machine inside an AWS AZ and VPC. It solves “give me a machine.” It does not automatically solve deployment, patching, application health, replication, or durable business data.

Mental picture: assemble a computer from six choices

    flowchart TB
        AMI["AMI
What disk image boots?"]
        TYPE["Instance type
How much CPU/memory/network?"]
        SUBNET["Subnet
Which VPC and AZ?"]
        ENI["Network interface
Which IP and security groups?"]
        DISK["EBS / instance store
Which storage lifecycle?"]
        ROLE["Instance profile
Which AWS permissions?"]
        EC2["EC2 instance"]
    
        AMI --> EC2
        TYPE --> EC2
        SUBNET --> ENI --> EC2
        DISK --> EC2
        ROLE --> EC2

Exact pieces

AMI versus a running instance

AMI = template/snapshot used to boot
    instance = a running virtual computer created from that template

Changing a running instance does not rewrite the original AMI.

    flowchart LR
        AMI["AMI v1"] --> A["Instance A"]
        AMI --> B["Instance B"]
        A -->|"manual package install"| A2["Instance A'"]
        B -.->|"unchanged"| B

Now A and B drift. A replacement created from AMI v1 resembles B, not manually modified A.

EC2 states are machine states

running does not mean GET /health/ready returns 200.

Storage: instance store, EBS, and S3 are different

    flowchart TB
        APP["Receipt API"]
        TEMP["Instance store / ephemeral disk
fast scratch, loss tolerated"]
        EBS["EBS
block device in one AZ"]
        S3["S3
regional object API"]
    
        APP -->|"temporary conversion file"| TEMP
        APP -->|"OS and local filesystem"| EBS
        APP -->|"durable receipt object"| S3

EBS can persist independently, depending on DeleteOnTermination, but it belongs to an AZ and behaves like a block device. S3 is not a mounted hard drive; it has object semantics.

Bootstrap with user data

A learning bootstrap:

#!/usr/bin/env bash
    set -euo pipefail
    
    dnf install -y nginx
    printf 'receipt-api learning host\n' >/usr/share/nginx/html/index.html
    systemctl enable --now nginx

Mental movie:

instance boots
    → cloud-init reads user data
    → package manager downloads nginx
    → file is written
    → service starts

Where can this fail?

• package repository unreachable because private subnet has no egress;
• package name/version changes;
• user-data command fails;
• the script includes a secret that appears in metadata/state/logs;
• service listens on a port the security group does not allow;
• repeated execution is not safe.

Production principle:

bake stable dependencies into an AMI or container image
    keep bootstrap short, idempotent, and observable
    retrieve secrets at runtime using a role

Give the instance an AWS identity

    sequenceDiagram
        participant A as Application on EC2
        participant M as Instance Metadata Service
        participant STS as AWS STS
        participant S3 as S3
    
        A->>M: SDK credential request
        M-->>A: Temporary instance-role credentials
        A->>S3: Signed API request
        S3-->>A: Result

An instance profile is the mechanism used to attach an IAM role to EC2. Require IMDSv2 for new instances. Do not let untrusted code/containers inherit a broad node role merely because they share the instance.

How should an operator connect?

Old habit:

public IP + port 22 open + shared SSH key

Better default to evaluate:

Systems Manager Session Manager
    → IAM-authorized session
    → no inbound SSH port required
    → private instance possible
    → auditable session controls

Session Manager still needs:

• SSM Agent;
• instance IAM permissions;
• network path to Systems Manager endpoints;
• operator IAM authorization.

The most mature operation is often not logging into the machine at all. Use automated deployment, metrics, and logs first.

One instance is not a service

    flowchart LR
        USER["Users"] --> ONE["One EC2 instance"]
        FAIL["Instance or AZ fails"] --> ONE
        ONE --> DOWN["All traffic stops"]

An Auto Scaling group (ASG) changes the model:

    flowchart TB
        LT["Launch template
AMI + type + role + network"]
        ASG["Auto Scaling group
min / desired / max"]
        A["Instance A · AZ A"]
        B["Instance B · AZ B"]
        HEALTH["EC2 and load-balancer health"]
    
        LT --> ASG
        ASG --> A
        ASG --> B
        HEALTH --> ASG
        ASG -->|"replace unhealthy"| NEW["Replacement instance"]

Two different ideas:

• desired capacity: keep N instances alive;
• dynamic scaling: change N based on demand/schedule.

An ASG can replace an unhealthy instance. It cannot repair a broken application release that every replacement boots.

Trace it — boot to healthy service

1. EC2 API receives RunInstances/ASG launch request.
    2. IAM authorizes the caller to launch.
    3. EC2 chooses capacity in the requested AZ/subnet.
    4. ENI receives a private IP and security groups.
    5. EBS root volume is created from the AMI.
    6. operating system boots.
    7. user data runs.
    8. application starts on 0.0.0.0:8080.
    9. health checker connects through the permitted path.
    10. application returns 200.

At step 10 the machine and application are both usable. Stopping at step 6 is not enough.

Break it

Prediction

The EC2 system status check passes. Does that prove users can upload receipts?

Can an EBS volume in AZ A be attached normally to an EC2 instance in AZ B?

Lab 3 — prove replacement

Build a learning instance through a launch template or Terraform. Then:

1. retrieve its instance ID and private IP;
2. verify the health endpoint;
3. make no manual application changes;
4. terminate it through the ASG;
5. watch replacement;
6. verify the new instance becomes healthy;
7. confirm its instance ID changed;
8. confirm the service behavior did not.

Completion criterion:

You trust the definition more than the individual machine.

Feel the problem first

If receipts live at:

/opt/receipt-api/data/abc.pdf

then replacing the instance may remove them. Scaling creates a second problem:

request writes to instance A
    later read reaches instance B
    instance B does not have the file

We need storage independent from any one compute instance.

Mental picture: a warehouse of sealed packages

Exact model:

bucket: receipt-prod-data
    key:    receipts/2026/07/23/abc.pdf
    value:  PDF bytes

receipts/2026/07/23/ looks like folders in a console, but it is a key prefix.

Object storage semantics

S3 is not POSIX:

• objects are addressed by complete keys;
• normal in-place byte edits are not the model;
• rename is generally copy plus delete;
• directory locks and filesystem transactions do not appear;
• clients call APIs such as PutObject, GetObject, and DeleteObject.

This is why the application should use an S3 SDK, not pretend the bucket is a normal disk without understanding the semantic trade-off.

Trace it — upload one receipt

    sequenceDiagram
        participant U as User
        participant API as Receipt API
        participant C as Credential provider
        participant S3 as S3
    
        U->>API: POST receipt.pdf
        API->>API: Authenticate user and validate PDF
        API->>C: Resolve temporary AWS credentials
        C-->>API: Receipt workload role session
        API->>S3: PutObject receipts/abc.pdf
        S3->>S3: Evaluate IAM, bucket, endpoint, KMS controls
        S3-->>API: Success, ETag/version metadata
        API-->>U: 201 Created, receipt ID abc

Application user authentication and IAM are different:

• the Receipt API decides whether this human may upload;
• IAM decides whether the Receipt workload may call S3.

Bucket permission versus object permission

arn:aws:s3:::receipt-prod-data

identifies the bucket.

arn:aws:s3:::receipt-prod-data/receipts/*

identifies objects under the prefix.

Examples:

Do not use s3: on because object and bucket scopes feel confusing. Read the API's authorization reference.

Keep it private

For receipt data:

• enable S3 Block Public Access;
• use IAM/bucket/access-point policy;
• avoid object ACL-based designs unless a specific compatibility case demands them;
• grant the workload prefix-level access;
• separate operator metadata access from receipt-content access.

“The bucket URL is hard to guess” is not access control.

Presigned URLs

Maybe a user needs to download a receipt without your API streaming all bytes.

    sequenceDiagram
        participant U as User
        participant API as Receipt API
        participant S3 as S3
    
        U->>API: GET /receipts/abc/download
        API->>API: Authorize this user for abc
        API-->>U: Short-lived presigned S3 URL
        U->>S3: GET using signed URL
        S3-->>U: PDF bytes

The bucket remains private. The URL temporarily delegates one operation using the signer's permissions.

Treat the URL like a bearer credential until expiration:

• short lifetime;
• HTTPS;
• do not log the full URL;
• authorize before generating it.

Versioning and lifecycle

    flowchart LR
        PUT1["Put abc.pdf v1"] --> V1["Version 1"]
        PUT2["Overwrite abc.pdf"] --> V2["Version 2 current"]
        DEL["Delete abc.pdf"] --> MARK["Delete marker current"]
        V1 --> HISTORY["Older versions retained"]
        V2 --> HISTORY

Versioning helps recover accidental overwrites/deletes. It also means:

• old versions consume storage;
• deleting the current view may not delete every version;
• lifecycle rules must account for non-current versions;
• a highly privileged attacker may still delete versions;
• recovery must be tested.

Lifecycle examples:

after 30 days → transition older receipts to a cheaper storage class
    after required retention → expire according to policy
    abort incomplete multipart uploads after a short window

Do not copy lifecycle numbers without a business retention rule.

Encryption has two authorization surfaces

With S3-managed encryption, S3 manages the keying layer.

With a customer-managed KMS key:

    flowchart LR
        APP["Receipt role"] --> S3P["S3 permission"]
        APP --> KMSP["KMS key permission"]
        S3P --> WRITE["Encrypted object write"]
        KMSP --> WRITE

Correct S3 permission plus missing KMS permission can still produce AccessDenied.

Consistency

S3 gives strong read-after-write consistency for object PUT and DELETE operations. After a successful new PutObject, an immediate S3 read can observe it.

That does not make every surrounding system instantaneous:

• a CDN may cache;
• your own database/index may lag;
• DNS and control-plane changes have other behavior;
• retries may create application-level duplicates.

Idempotency

If the client retries POST /receipts after a timeout, did S3 store the object the first time? The client may not know.

A robust API can accept an idempotency key:

same user + same idempotency key
    → same logical receipt result
    → no duplicate business action

S3 object keys can be designed from stable request IDs rather than random new IDs on every retry.

Break it

Prediction

Does the key receipts/2026/abc.pdf prove two directories exist?

The pod successfully writes an object using a KMS key but cannot read it. What extra permission might be missing?

Lab 4 — make storage independent from compute

1. create a private learning bucket with versioning;
2. use a temporary-role session;
3. upload receipts/test.txt;
4. replace its content;
5. list versions;
6. delete the current object;
7. restore/read an older version;
8. verify public access fails;
9. inspect the exact caller in CloudTrail if data events are configured;
10. remove all test versions intentionally during teardown.

Completion criterion:

You can replace the compute host without moving the receipt object.

Feel the problem first

Docker solved packaging:

application + runtime + dependencies → image

But production needs:

• a trusted place to push it;
• permissions for publishers and runtimes;
• an exact version identity;
• vulnerability scanning;
• retention/cleanup;
• optional cross-account distribution.

ECR is the registry. It stores the image; it does not run it.

Mental picture: a library

Registry, repository, image

    flowchart TB
        REG["Private ECR registry
account + Region"]
        REPO1["Repository: receipt-api"]
        REPO2["Repository: worker"]
        IMG1["Image digest sha256:aaa..."]
        IMG2["Image digest sha256:bbb..."]
    
        REG --> REPO1
        REG --> REPO2
        REPO1 --> IMG1
        REPO1 --> IMG2

Repository URI:

123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api

Tag versus digest

    flowchart LR
        TAG["tag: main"] --> OLD["sha256:111..."]
        PUSH["push new content to mutable main"] --> TAG
        TAG -.-> NEW["sha256:222..."]

A tag is a pointer. A digest identifies content.

These two commands may not mean the same bytes tomorrow:

receipt-api:latest
    receipt-api:main

This does:

receipt-api@sha256:222...

Production rule:

tag for humans
    digest for deployment truth

Build once, promote the same bytes

Bad:

    flowchart LR
        COMMIT["Commit abc"] --> DEV["Build for dev"]
        COMMIT --> STAGE["Later rebuild for staging"]
        COMMIT --> PROD["Later rebuild for prod"]

Package repositories and base-image tags can change. These may be three different artifacts.

Better:

    flowchart LR
        COMMIT["Commit abc"] --> BUILD["Build once"]
        BUILD --> DIGEST["sha256:222..."]
        DIGEST --> DEV["Dev"]
        DIGEST --> STAGE["Staging"]
        DIGEST --> PROD["Production"]

Environment configuration stays outside the image.

Push flow

    sequenceDiagram
        participant CI as GitHub Actions
        participant STS as AWS STS
        participant ECR as ECR
    
        CI->>STS: OIDC token → assume publisher role
        STS-->>CI: Temporary credentials
        CI->>ECR: Authenticate registry client
        CI->>ECR: Check/upload layers
        CI->>ECR: Upload manifest and tag
        ECR-->>CI: Image digest

Publisher permissions and runtime pull permissions are different.

Who pulls?

Image pull occurs before application code starts. A pod cannot use its runtime role to fix an image that never started.

Tag immutability

If tag immutability is enabled:

push git-abc → digest 111 succeeds
    push different content to git-abc → rejected

This makes a tag a safer human label, though the digest remains the strongest content identity.

Scanning

ECR scanning can find known vulnerabilities in packages/layers.

It cannot prove:

• business logic is correct;
• authorization is correct;
• no secret was baked into a layer;
• runtime configuration is safe;
• a vulnerability database will never update tomorrow.

Treat scanning as:

finding → policy decision → exception/remediation owner → re-scan

Lifecycle policy

The registry will grow forever without a policy. But deletion can remove rollback artifacts.

Ask:

• how many releases must be instantly recoverable?
• how long do incident investigations need old digests?
• can a running cluster pull a digest again after node replacement?
• are images replicated to another account/Region?

Do not immediately delete all untagged images: a digest can be deployed even when no tag points to it.

Break it

Prediction

Two pods both display receipt-api:main. Must they run identical content?

Can ECR scanning tell whether /receipts/{id} has a broken authorization check?

Lab 5 — prove content identity

1. build receipt-api:git-abc;
2. push it to a learning ECR repository;
3. record the digest;
4. pull with repository@digest;
5. rebuild after changing one file;
6. compare the new digest;
7. attempt to reuse an immutable tag;
8. inspect scan results;
9. explain which identity pushed and which identity pulled.

Completion criterion:

You can answer “which exact bytes ran?” with a digest.

Feel the problem first

A user says:

Receipt upload is slow.

Without telemetry, you can only guess:

• DNS?
• ALB?
• one unhealthy target?
• CPU?
• S3 latency?
• IAM retries?
• a new deployment?
• a full disk?

CloudWatch is where many AWS metrics, logs, alarms, and dashboards live. It does not automatically know what “Receipt API is successful” means. We must instrument that.

Mental picture: a car dashboard plus flight recorder

Metrics are time series

Metric identity:

namespace + metric name + dimension set

Example:

namespace: AWS/ApplicationELB
    metric:    TargetResponseTime
    dimensions:
      LoadBalancer=app/receipt/...
      TargetGroup=targetgroup/receipt/...

A different dimension set is a different time series.

“CPU is 80%” is incomplete

Ask:

Which resource?
    Which Region?
    Which period?
    Average, maximum, sum, or percentile?
    For how long?
    Compared with what traffic?

An average can hide one saturated target. A one-second maximum can exaggerate noise.

Logs should be structured evidence

Better:

{
      "timestamp": "2026-07-23T14:02:11Z",
      "level": "error",
      "service": "receipt-api",
      "version": "sha256:222...",
      "request_id": "req-7f1",
      "route": "POST /receipts",
      "duration_ms": 843,
      "error_code": "receipt_write_failed"
    }

Worse:

something went wrong

Structured fields let you ask:

show errors
    for POST /receipts
    after digest 222 began
    grouped by error_code

Never log:

• AWS credentials or session tokens;
• passwords;
• full presigned URLs;
• receipt contents or sensitive personal data;
• unbounded request bodies.

Log groups, streams, and retention

    flowchart TB
        GROUP["Log group
/receipt/prod/api
retention + access policy"]
        S1["Stream
pod A / container"]
        S2["Stream
pod B / container"]
        E1["Event: request completed"]
        E2["Event: S3 AccessDenied"]
        E3["Event: request completed"]
    
        GROUP --> S1
        GROUP --> S2
        S1 --> E1
        S1 --> E2
        S2 --> E3

Set retention deliberately. “Never delete logs” is not free and may violate data governance.

CloudWatch does not collect every EC2 file or EKS container log simply because the resource exists. You need an agent/log driver/collector path and permissions.

Alarms evaluate metrics

    flowchart LR
        METRIC["5xx error rate"] --> RULE["Over threshold
for N evaluation periods"]
        RULE --> STATE["OK / ALARM / INSUFFICIENT_DATA"]
        STATE --> ACTION["Notify/on-call/automation"]
        ACTION --> RUNBOOK["Runbook with first checks"]

A useful alarm has:

• user or system consequence;
• threshold and period justified by behavior;
• intentional missing-data treatment;
• owner;
• notification path;
• runbook;
• tested delivery.

An alarm that no one receives is a database row, not protection.

Start with user symptoms, then causes

Symptom signals

high request error rate
    high p95/p99 latency
    zero successful receipt uploads
    zero healthy targets

Cause signals

CPU/memory saturation
    pod restarts
    S3 AccessDenied
    image pull failures
    subnet IP exhaustion
    disk full

If you alert only on CPU, an IAM outage can break every request while CPU remains low.

Four golden signals

Add a business signal:

valid receipts durably stored per minute

An HTTP 201 followed by a corrupted object would look technically successful but fail the business outcome.

CloudWatch versus CloudTrail

CloudWatch is operational evidence. CloudTrail is AWS API/account activity evidence. An incident timeline often uses both.

Trace it — one slow request

request_id req-7f1
    → ALB target response time: 843 ms
    → app log total: 820 ms
    → app dependency field s3_ms: 780 ms
    → S3 call succeeded
    → CPU only 20%

This evidence points at dependency latency, not CPU scaling.

Now:

ALB time: 843 ms
    app log: no matching request
    target connection error

The problem is before application handling—target/network/health—not S3.

Break it

Prediction

ALB has zero healthy targets. Is average EC2 CPU the first graph to inspect?

A deployment is Synced in Argo CD. Does that prove the application SLO is healthy?

Lab 6 — create an evidence chain

Send five successful uploads and one intentionally invalid upload.

Prove:

1. ALB/request metric count changed;
2. application logs contain six request IDs;
3. invalid input is a controlled 4xx, not a 5xx;
4. successful logs name the deployed digest;
5. a query can group outcomes by status/error code;
6. no receipt content or credentials appear;
7. log retention is explicit;
8. an alarm test reaches the expected notification path.

Completion criterion:

You can begin with a user symptom and locate the responsible layer using evidence.

Feel the problem first

We now have two application copies:

10.20.10.31:8080
    10.20.11.42:8080

Users should not:

• remember those addresses;
• choose which copy to call;
• know when a copy is replaced;
• send traffic to an unhealthy process;
• update bookmarks when addresses change.

We need two different things:

1. a stable name;
2. a traffic entry point that selects healthy application targets.

Route 53 supplies DNS. An Application Load Balancer supplies HTTP/HTTPS routing.

Mental picture: directory plus receptionist

DNS does not carry the user's HTTP conversation. It helps the client find the ALB.

DNS resolution first

    sequenceDiagram
        participant B as Browser
        participant R as Recursive resolver
        participant D as Route 53 authoritative DNS
        participant A as ALB
    
        B->>R: What is api.example.com?
        R->>D: DNS query
        D-->>R: Alias answer for ALB
        R-->>B: Resolved ALB addresses
        B->>A: Separate HTTPS connection

The DNS lookup may be cached. The later HTTPS request goes to the ALB, not “through Route 53.”

Hosted zones and records

A hosted zone contains DNS records for a domain.

For the Receipt API:

api.example.com
    A alias
    → receipt-alb-123.eu-west-2.elb.amazonaws.com

An alias can work at the zone apex and tracks the changing addresses of the supported AWS target.

The DNS delegation chain

Owning a hosted zone in Route 53 is not sufficient if the domain's registrar delegates to different authoritative name servers.

    flowchart LR
        ROOT["DNS root"] --> TLD[".com name servers"]
        TLD --> AUTH["example.com authoritative
Route 53 name servers"]
        AUTH --> RECORD["api.example.com alias"]
        RECORD --> ALB["ALB"]

If delegation is wrong, the correct record can exist in a zone nobody asks.

TTL and change

Resolvers cache DNS answers. A DNS update cannot recall already cached answers.

Therefore:

• lower TTL before a planned migration when you control TTL;
• wait for old cache lifetimes;
• keep the old endpoint healthy during transition;
• remember alias TTL behavior follows the AWS target.

DNS failover is not instantaneous connection migration. Existing connections and cached answers have their own lifetimes.

ALB has four essential layers

    flowchart LR
        CLIENT["Client"] --> LB["1 · Load balancer
subnets + security group"]
        LB --> LISTENER["2 · Listener
HTTPS :443"]
        LISTENER --> RULE["3 · Ordered rules
host/path → action"]
        RULE --> TG["4 · Target group
targets + health check"]
        TG --> A["Target A"]
        TG --> B["Target B"]

1. Load balancer

The regional entry point. An internet-facing ALB uses public subnets in selected AZs. Its security group accepts intended client traffic.

2. Listener

Accepts a protocol/port:

HTTP :80  → redirect to HTTPS
    HTTPS:443 → evaluate application rules

3. Listener rules

Examples:

host api.example.com + path /receipts/* → receipt target group
    host admin.example.com                  → admin target group
    default                                 → fixed 404

Rules are ordered. A broad earlier rule can capture traffic before a later specific rule.

4. Target group

Contains registered instance IDs or IP addresses plus:

• target protocol and port;
• health-check path/port;
• healthy/unhealthy thresholds;
• deregistration delay;
• routing attributes.

Layer 7 means HTTP awareness

ALB understands HTTP/HTTPS concepts: host, path, headers, methods, redirects, and responses.

For raw TCP/UDP or different network-layer requirements, evaluate a Network Load Balancer. “A load balancer” is not one universal interchangeable object.

TLS termination

    flowchart LR
        CLIENT["Client"] -->|"HTTPS :443"| ALB["ALB + ACM certificate"]
        ALB -->|"HTTP or HTTPS
according to design"| APP["Receipt target"]

Common model:

• ACM certificate covers api.example.com;
• certificate is in the ALB's Region;
• port 80 redirects;
• port 443 uses an approved TLS policy;
• backend encryption is chosen according to threat/compliance requirements.

The hostname is part of certificate validation. Pointing a new name at the same ALB does not make the certificate valid for that new name.

Health checks: “may I send user traffic?”

Good readiness endpoint:

GET /health/ready
    
    HTTP/1.1 200 OK

Properties:

• fast;
• no side effect;
• no end-user authentication;
• no sensitive diagnostics;
• listens on the real application port;
• represents ability to serve traffic now.

Do not answer 200 simply because the process exists. Do not make readiness depend on every optional dependency and create cascading removal of all targets.

Liveness versus readiness

ALB target health is closest to readiness.

Security-group chain

ALB SG:
      inbound 443 from intended clients
      outbound 8080 to app SG
    
    App SG:
      inbound 8080 from ALB SG

The application remains private even though the service is public.

Trace it — one successful request

1. Browser resolves api.example.com.
    2. Route 53 returns ALB addresses through the alias.
    3. Browser opens TLS connection to ALB:443.
    4. ALB presents the certificate.
    5. listener evaluates host/path rules.
    6. matching rule selects receipt target group.
    7. ALB selects one healthy target.
    8. ALB security group and app security group permit port 8080.
    9. Receipt process handles the request.
    10. response returns through ALB to browser.

Break it — read the symptom from outside inward

Target-timeout decision tree

    flowchart TD
        START["Target health = timeout"] --> REG{"Target registered
with correct IP:port?"}
        REG -->|"no"| FIXREG["Fix Service/controller/target registration"]
        REG -->|"yes"| SG{"App SG allows
source ALB SG?"}
        SG -->|"no"| FIXSG["Fix SG rule"]
        SG -->|"yes"| LISTEN{"Process listens on
0.0.0.0:targetPort?"}
        LISTEN -->|"no"| FIXAPP["Fix bind address/port"]
        LISTEN -->|"yes"| PATH{"Health path responds
within timeout?"}
        PATH -->|"no"| FIXPATH["Fix endpoint/startup/dependency"]
        PATH -->|"yes"| DEEP["Inspect NACL/network policy
and target-health details"]

Prediction

curl https://api.example.com gets an ALB-generated 404. Should you change Route 53?

A pod works through kubectl port-forward but is unhealthy behind the ALB. Does port-forward prove the ALB network path?

Lab 7 — follow the name

For a learning endpoint:

dig api.example.com
    curl --verbose https://api.example.com/health/ready
    aws elbv2 describe-target-health --target-group-arn "$TARGET_GROUP_ARN"

Then intentionally:

1. use a wrong health-check path;
2. observe the target reason;
3. restore the path;
4. wait for healthy threshold;
5. change the listener default response;
6. observe the difference between ALB response and application response.

Completion criterion:

From an HTTP symptom, you can state which earlier layers already succeeded.

Feel the problem first

We can run a container:

docker run -p 8080:8080 receipt-api

Production asks:

• who starts it if it exits?
• how many copies should run?
• which subnets and security groups?
• who pulls the ECR image?
• which AWS permissions does the application get?
• how does it join the ALB target group?
• how does a new version roll out?
• where do logs go?

ECS is an AWS-native container orchestrator that answers those scheduling and lifecycle questions.

Mental picture: recipe, meal, restaurant manager

Exact model:

    flowchart TB
        TD["Task definition revision
image + CPU/memory + command + roles + logs"]
        TASK["Task
running instance of task definition"]
        SERVICE["Service
desired count + deployment"]
        CLUSTER["Cluster
logical scheduling boundary"]
        CAPACITY["Fargate or EC2 capacity"]
    
        TD --> TASK
        TD --> SERVICE
        SERVICE --> TASK
        CLUSTER --> SERVICE
        CAPACITY --> TASK

Task definition

An illustrative definition:

{
      "family": "receipt-api",
      "networkMode": "awsvpc",
      "requiresCompatibilities": ["FARGATE"],
      "cpu": "512",
      "memory": "1024",
      "executionRoleArn": "arn:aws:iam::123456789012:role/receipt-ecs-execution",
      "taskRoleArn": "arn:aws:iam::123456789012:role/receipt-runtime",
      "containerDefinitions": [
        {
          "name": "api",
          "image": "123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api@sha256:222...",
          "essential": true,
          "portMappings": [
            {
              "containerPort": 8080,
              "protocol": "tcp"
            }
          ],
          "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
              "awslogs-group": "/receipt/prod/api",
              "awslogs-region": "eu-west-2",
              "awslogs-stream-prefix": "ecs"
            }
          }
        }
      ]
    }

It describes a blueprint. A task is the running copy.

Task definitions are revisioned:

receipt-api:17
    receipt-api:18

The service moves from one revision to another.

Fargate versus EC2

    flowchart TB
        ECS["ECS scheduler"]
        F["Fargate
AWS abstracts worker hosts"]
        E["ECS on EC2
you manage cluster instances"]
        T1["Tasks"]
        T2["Tasks"]
    
        ECS --> F --> T1
        ECS --> E --> T2

Fargate removes node administration. It does not remove VPC, IAM, health, task sizing, logs, or deployment design.

awsvpc networking

Each task receives an ENI and VPC address:

    flowchart LR
        ALB["ALB"] -->|"10.20.10.31:8080"| ENI1["Task ENI A"]
        ALB -->|"10.20.11.42:8080"| ENI2["Task ENI B"]
        ENI1 --> T1["Receipt task"]
        ENI2 --> T2["Receipt task"]

Consequences:

• choose subnets and security groups per service;
• target group can register task IPs;
• subnet IP capacity limits tasks;
• rollout surge needs temporary extra addresses;
• private-subnet egress must reach ECR/log/S3 endpoints.

Two roles: execution and task

    flowchart TB
        ECS["ECS runtime"] -->|"Execution role"| START["Pull ECR image
start task
deliver awslogs"]
        APP["Receipt code"] -->|"Task role"| DATA["Put receipt in S3"]

This is a frequent interview and production-debugging distinction.

Service desired state

desired count = 3
    running healthy tasks = 2
    → service scheduler starts another task

The service also owns a rollout:

    sequenceDiagram
        participant S as ECS service
        participant R as Runtime
        participant A as ALB
    
        S->>R: Start task definition revision 18
        R->>R: Pull image using execution role
        R->>R: Create task ENI and start container
        R->>A: Register target
        A-->>S: Health checks pass
        S->>A: Drain old revision 17 target
        S->>R: Stop old task

If the new target never becomes healthy, deployment parameters and circuit-breaker behavior determine what happens next.

Autoscaling

ECS service autoscaling changes desired task count based on signals:

ALB requests per target rises
    → desired task count increases
    → scheduler launches tasks
    → tasks become healthy
    → traffic spreads across more targets

Scaling cannot fix:

• image pull denied;
• broken application version;
• no free subnet IPs;
• hard service quota;
• database/dependency bottleneck.

Trace it — why a task never stays running

Read the service event and stopped-task reason:

    flowchart TD
        START["Task fails"] --> PLACE{"Could scheduler place it?"}
        PLACE -->|"no"| CAP["Capacity/subnet/quota"]
        PLACE -->|"yes"| PULL{"Could runtime pull image?"}
        PULL -->|"no"| ECR["Execution role / ECR / egress"]
        PULL -->|"yes"| RUN{"Did container process stay up?"}
        RUN -->|"no"| LOG["Command/config/app logs"]
        RUN -->|"yes"| HEALTH{"Did ALB health pass?"}
        HEALTH -->|"no"| NET["Port/path/SG/grace period"]
        HEALTH -->|"yes"| OK["Healthy service task"]

Do not increase desired count before finding which gate failed.

Break it

Prediction

Desired count is three; all tasks immediately stop. Should autoscaling create six?

The execution role can write receipts to S3. Is that the right design?

ECS or EKS yet?

At this stage, ECS is enough when:

• the AWS-native object model meets team needs;
• Kubernetes APIs/ecosystem are not requirements;
• smaller platform surface is valuable.

Do not migrate to EKS because it sounds more advanced. Add it when the Kubernetes model solves a real organizational or technical need.

Lab 8 — four-gate deployment

Deploy two Fargate tasks behind an ALB. Prove separately:

1. placement succeeded;
2. ECR pull succeeded;
3. container stayed running;
4. target became healthy;
5. S3 runtime call succeeded.

Break one gate at a time. Record the exact service event/log/metric that distinguishes it.

Completion criterion:

You never use “ECS is broken” as a diagnosis.

Feel the problem first

Organizations choose Kubernetes when they want a common declarative application platform with:

• Kubernetes workload APIs;
• a large controller/operator ecosystem;
• consistent scheduling and deployment abstractions;
• portability of workload specifications;
• shared platform conventions across teams.

The cost is another distributed system and more concepts to operate.

EKS lets AWS operate the Kubernetes control plane. You still operate cluster configuration, compute, workloads, add-ons, identity, network policy, upgrades, and observability.

First understand Kubernetes in one sentence

Kubernetes continuously tries to make the actual cluster state match the desired state stored through its API.

Example:

desired: 3 Receipt pods
    actual:  2 Ready pods
    gap:     1 pod
    action:  create/schedule another pod

Mental picture: thermostat, not installer

    flowchart TD
        DESIRED["Desired state
Deployment replicas: 3"]
        OBSERVE["Controller observes
Ready replicas: 2"]
        DIFF["Difference: one missing"]
        ACT["Create replacement"]
        REPEAT["Observe again"]
    
        DESIRED --> OBSERVE --> DIFF --> ACT --> REPEAT --> OBSERVE

The loop does not run once. Reconciliation is the platform's normal behavior.

EKS control plane versus data plane

    flowchart TB
        subgraph AWSMANAGED["AWS-managed EKS control plane"]
            API["Kubernetes API server"]
            ETCD["etcd desired/cluster state"]
            CTRL["scheduler + controllers"]
        end
    
        subgraph YOURACCOUNT["Your account / VPC data plane"]
            N1["Worker node A"]
            N2["Worker node B"]
            P1["Receipt pod A"]
            P2["Receipt pod B"]
            ADD["CNI, DNS, proxies, CSI,
controllers, agents"]
        end
    
        API <--> N1
        API <--> N2
        N1 --> P1
        N2 --> P2
        ADD <--> API

AWS maintains control-plane availability and replacement. That does not spread your pods or create worker capacity automatically.

The objects you need

    flowchart TB
        DEP["Deployment
replicas + pod template + rollout"]
        RS["ReplicaSet
one rollout revision"]
        POD["Pod
one or more co-scheduled containers"]
        SVC["Service
stable in-cluster endpoint"]
        ING["Ingress
HTTP routing intent"]
        SA["ServiceAccount
Kubernetes workload identity"]
        CM["ConfigMap
non-secret configuration"]
    
        DEP --> RS --> POD
        SVC --> POD
        ING --> SVC
        SA --> POD
        CM --> POD

Pod

Smallest scheduled unit. Containers in one pod share network namespace and lifecycle coupling. A pod is replaceable.

Deployment

States desired replicas and rollout strategy. It creates ReplicaSets.

Service

Gives changing pods a stable in-cluster virtual endpoint selected by labels.

Ingress

Declares HTTP routing. An Ingress controller must translate it into working data-plane infrastructure.

ServiceAccount

The pod's Kubernetes identity. It can also be associated with an AWS IAM role through EKS Pod Identity.

A minimal Receipt Deployment

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: receipt-api
      namespace: receipt
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: receipt-api
      template:
        metadata:
          labels:
            app: receipt-api
        spec:
          serviceAccountName: receipt-api
          containers:
            - name: api
              image: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api@sha256:222...
              ports:
                - name: http
                  containerPort: 8080
              resources:
                requests:
                  cpu: 200m
                  memory: 256Mi
                limits:
                  memory: 512Mi
              readinessProbe:
                httpGet:
                  path: /health/ready
                  port: http
                periodSeconds: 5

Read it as:

Keep three pods matching this template. Use this exact image digest, service account, port, resource contract, and readiness check.

Labels and selectors form relationships

    flowchart LR
        SVC["Service selector
app=receipt-api"] --> P1["Pod label
app=receipt-api"]
        SVC --> P2["Pod label
app=receipt-api"]
        WRONG["Pod label
app=receipt"] -.->|"not selected"| SVC

A Service with zero endpoints can exist perfectly. The object is valid; its selector matches no Ready pods.

Three identity doors

    flowchart TB
        HUMAN["Human / CI AWS role"] --> AWSAUTH["1 · AWS authentication
obtain EKS token"]
        AWSAUTH --> CLUSTER["2 · EKS access entry +
Kubernetes RBAC"]
        CLUSTER --> KAPI["Kubernetes API action"]
    
        POD["Receipt pod + ServiceAccount"] --> PODID["3 · EKS Pod Identity"]
        PODID --> S3["AWS S3 API"]

Door 1: AWS authentication

Can the IAM principal authenticate for the EKS cluster?

Door 2: cluster authorization

Does an EKS access entry/access policy or Kubernetes RBAC permit get pods, apply Deployment, or delete node?

Door 3: pod-to-AWS authorization

May Receipt application code call s3:PutObject?

These are separate. Cluster administrators should not automatically become S3 data administrators. A pod S3 role should not become Kubernetes admin.

EKS Pod Identity

Association:

cluster:        receipt-prod
    namespace:      receipt
    serviceAccount: receipt-api
    IAM role:       receipt-prod-s3-writer

    sequenceDiagram
        participant P as Pod
        participant SDK as AWS SDK
        participant A as Pod Identity Agent
        participant E as EKS Auth/STS path
        participant S as S3
    
        P->>SDK: PutObject
        SDK->>A: Resolve credentials
        A->>E: Obtain associated role credentials
        E-->>SDK: Temporary session
        SDK->>S: Signed PutObject

The cluster needs the agent unless its compute mode supplies the functionality. The application needs a supported SDK/default credential chain.

IRSA is another EKS workload-identity mechanism. Pod Identity is the current default to evaluate for ordinary EKS scenarios; choose based on supported compute, cross-account, and environment needs.

Pod networking with the VPC CNI

On EC2 worker nodes, the Amazon VPC CNI gives pods VPC addresses.

    flowchart TB
        SUBNET["Private subnet 10.20.10.0/24"]
        ENI["Node ENIs + IP/prefix pool"]
        P1["Pod 10.20.10.31"]
        P2["Pod 10.20.10.32"]
        P3["Pod 10.20.10.33"]
    
        SUBNET --> ENI
        ENI --> P1
        ENI --> P2
        ENI --> P3

This is why:

• node instance type affects pod IP capacity;
• subnet size affects pod scaling;
• warm addresses consume capacity;
• rolling surge needs extra IPs;
• VPC routes/security can reach pod IPs;
• “nodes have CPU” does not prove a new pod can get an IP.

How Ingress becomes an ALB

An Ingress is intent. The AWS Load Balancer Controller performs the AWS work.

    sequenceDiagram
        participant G as Git/Argo CD
        participant K as Kubernetes API
        participant C as AWS Load Balancer Controller
        participant E as ELBv2 APIs
    
        G->>K: Apply Ingress
        C->>K: Observe Ingress
        C->>E: Create/update ALB, rules, target group
        E-->>C: AWS resource status
        C->>K: Update Ingress status

The controller needs:

• correct Ingress class/annotations;
• discoverable/configured subnets;
• an IAM workload role;
• AWS API/network access;
• compatible controller version.

If Ingress exists but no ALB exists, inspect this controller rather than Route 53.

Illustrative Ingress:

apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: receipt-api
      namespace: receipt
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/healthcheck-path: /health/ready
    spec:
      ingressClassName: alb
      rules:
        - host: api.example.com
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: receipt-api
                    port:
                      number: 80

Probes

startupProbe:
      httpGet:
        path: /health/live
        port: http
      failureThreshold: 30
      periodSeconds: 2
    
    readinessProbe:
      httpGet:
        path: /health/ready
        port: http
      periodSeconds: 5
    
    livenessProbe:
      httpGet:
        path: /health/live
        port: http
      periodSeconds: 10

    flowchart TD
        START["Container starts"] --> SP{"Startup probe passed?"}
        SP -->|"not yet"| WAIT["Keep waiting within threshold"]
        SP -->|"yes"| READY{"Readiness passed?"}
        READY -->|"no"| NOTRAFFIC["Run but receive no Service traffic"]
        READY -->|"yes"| TRAFFIC["Receive traffic"]
        TRAFFIC --> LIVE{"Liveness continues passing?"}
        LIVE -->|"no"| RESTART["Restart container"]
        LIVE -->|"yes"| TRAFFIC

Do not make liveness call a fragile external dependency. An S3 incident should not necessarily cause every pod to restart continuously.

Scheduling and resource requests

The scheduler uses requests to find a node.

pod requests 200m CPU, 256Mi
    → scheduler finds node with at least that unallocated requested capacity

Memory limit:

process exceeds 512Mi
    → container can be OOM-killed

CPU and memory do not behave identically. CPU limits can throttle; memory exhaustion terminates.

Requests are not documentation. They drive placement and scaling.

Scaling is several control loops

    flowchart LR
        LOAD["Traffic rises"] --> HPA["HPA requests more pods"]
        HPA --> PEND["Pods pending if node capacity absent"]
        PEND --> NA["Node autoscaler/Karpenter adds nodes"]
        NA --> EC2["EC2 capacity joins"]
        EC2 --> SCHED["Scheduler places pods"]
        SCHED --> READY["Pods become Ready"]
        READY --> ALB["ALB target health passes"]

Each loop has its own signal and delay.

• HPA does not create nodes.
• Node autoscaling does not repair a bad image.
• Neither can add addresses to a full CIDR.
• ALB does not send traffic until targets are healthy.

Availability placement

Three replicas can land on one node/AZ unless constraints and capacity guide placement.

Use topology spread/anti-affinity where the availability requirement demands it. Provide node capacity across AZs.

A PodDisruptionBudget controls eviction-aware voluntary disruptions:

apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: receipt-api
      namespace: receipt
    spec:
      minAvailable: 2
      selector:
        matchLabels:
          app: receipt-api

It does not:

• prevent involuntary node/AZ failure;
• create replicas;
• guarantee a Deployment rollout cannot reduce availability;
• help when only one replica is healthy;
• replace topology spread.

Storage in EKS

EBS CSI creates/manages EBS volumes for Kubernetes claims. EBS remains AZ-scoped, so scheduling and storage topology must align.

ECS versus EKS

Kubernetes gives powerful abstractions. You pay in platform surface, upgrades, policy, networking, and cognitive load.

Break it

Prediction

The EKS cluster status is ACTIVE. Does that prove worker nodes and Receipt pods are healthy?

A developer may call eks:DescribeCluster but kubectl get pods is forbidden. Which door failed?

HPA asks for ten pods; six remain Pending. Does HPA itself need EC2 permission?

Lab 9 — watch reconciliation

kubectl -n receipt get deployment,replicaset,pods -w

Then:

1. delete one pod;
2. watch Deployment/ReplicaSet replace it;
3. change replicas from 3 to 5 through desired state;
4. watch scheduling and readiness;
5. change the Service selector so it matches nothing;
6. inspect EndpointSlices and ALB target health;
7. restore the selector through Git;
8. identify the new pod IPs.

Completion criterion:

You can name the controller that reacts to each change and the evidence it leaves.

Feel the problem first

Imagine we created the platform manually:

Alice creates VPC in console.
    Bob adds a route two days later.
    CI creates an ECR repository.
    Someone changes a security group during an incident.
    Staging was built from memory and differs from production.
    Nobody knows which changes are intentional.

A screenshot cannot answer:

• what should exist?
• what changed?
• who reviews the next change?
• how do we reproduce it?
• which tool owns this resource?
• which real object corresponds to this line of configuration?

Terraform gives us configuration plus state and a plan/apply workflow.

Mental picture: blueprint plus ledger

The ledger matters. Two visually identical buildings are not the same building. State records which real resource Terraform owns.

The exact comparison

    flowchart LR
        CONFIG["Configuration
what managed objects should be"]
        STATE["State
Terraform address ↔ AWS ID"]
        AWS["AWS APIs
what objects are now"]
        PLAN["Plan
actions to close the difference"]
    
        CONFIG --> PLAN
        STATE --> PLAN
        AWS --> PLAN
        PLAN -->|"approved apply"| AWS
        PLAN -->|"successful result"| STATE

Terraform needs all three inputs:

• configuration alone does not say which existing object it owns;
• state alone does not contain your desired design;
• remote APIs reveal drift and current attributes.

Terraform is not normally a continuous controller. It compares when a run occurs.

HCL: describe relationships, not a script

resource "aws_cloudwatch_log_group" "receipt_api" {
      name              = "/receipt/${var.environment}/api"
      retention_in_days = 30
    }

Read:

resource type: aws_cloudwatch_log_group
    local Terraform name: receipt_api
    desired attributes:
      name = /receipt/<environment>/api
      retention = 30 days

The Terraform address is:

aws_cloudwatch_log_group.receipt_api

It is not the AWS log-group name. It is Terraform's address inside this state.

Provider and version boundaries

terraform {
      required_version = "~> 1.15.0"
    
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 6.55"
        }
      }
    }
    
    provider "aws" {
      region = var.aws_region
    
      default_tags {
        tags = {
          Project     = "receipt"
          Environment = var.environment
          Owner       = "platform-team"
          ManagedBy   = "terraform"
        }
      }
    }

Mental separation:

Terraform CLI version ≠ AWS provider version

The provider is a separately versioned plugin that translates Terraform resource operations into AWS API calls.

The constraint is a reviewed compatibility choice. The dependency lock file records the selected provider build. Commit .terraform.lock.hcl.

Provider authentication

Terraform calls AWS APIs as an AWS principal.

Bad:

provider "aws" {
      access_key = "AKIA..."
      secret_key = "..."
    }

Good credential sources:

local human → federated AWS profile/SSO role
    GitHub workflow → OIDC temporary deployment role
    AWS-hosted runner → workload role

Before a plan, prove the context:

aws sts get-caller-identity
    aws configure list

Know the account, role, and Region. A perfectly valid plan in the wrong account is still wrong.

Variables, locals, data, resources, outputs

variable "environment" {
      type = string
    
      validation {
        condition     = contains(["dev", "stage", "prod"], var.environment)
        error_message = "environment must be dev, stage, or prod"
      }
    }
    
    locals {
      name_prefix = "receipt-${var.environment}"
    }
    
    data "aws_caller_identity" "current" {}
    
    resource "aws_ecr_repository" "receipt_api" {
      name                 = "${local.name_prefix}-api"
      image_tag_mutability = "IMMUTABLE"
    }
    
    output "repository_url" {
      value = aws_ecr_repository.receipt_api.repository_url
    }

An output marked sensitive is hidden in normal CLI display, but the value can still exist in state. Do not treat sensitive as encryption.

References create graph edges

resource "aws_iam_policy" "receipt_s3" {
      name   = "${local.name_prefix}-write-receipts"
      policy = data.aws_iam_policy_document.receipt_s3.json
    }

The reference tells Terraform the policy document must be known before creating the IAM policy.

    flowchart LR
        BUCKET["aws_s3_bucket.receipts"] --> DOC["data.aws_iam_policy_document.receipt_s3"]
        DOC --> POLICY["aws_iam_policy.receipt_s3"]
        POLICY --> ATTACH["role policy attachment"]

Prefer real value references over manual depends_on. Data flow teaches Terraform the dependency and supplies the value.

Build the S3 part

resource "aws_s3_bucket" "receipts" {
      bucket_prefix = "receipt-${var.environment}-data-"
    }
    
    resource "aws_s3_bucket_public_access_block" "receipts" {
      bucket = aws_s3_bucket.receipts.id
    
      block_public_acls       = true
      block_public_policy     = true
      ignore_public_acls      = true
      restrict_public_buckets = true
    }
    
    resource "aws_s3_bucket_versioning" "receipts" {
      bucket = aws_s3_bucket.receipts.id
    
      versioning_configuration {
        status = "Enabled"
      }
    }
    
    resource "aws_s3_bucket_server_side_encryption_configuration" "receipts" {
      bucket = aws_s3_bucket.receipts.id
    
      rule {
        apply_server_side_encryption_by_default {
          sse_algorithm = "AES256"
        }
      }
    }

Each resource owns one part of the bucket's configuration.

Now the policy uses the exact generated ARN:

data "aws_iam_policy_document" "receipt_s3" {
      statement {
        sid    = "WriteReceipts"
        effect = "Allow"
    
        actions = [
          "s3:PutObject",
          "s3:AbortMultipartUpload"
        ]
    
        resources = [
          "${aws_s3_bucket.receipts.arn}/receipts/*"
        ]
      }
    }

This is better than typing a guessed bucket ARN in three places.

Build the VPC mentally before using a module

    flowchart TB
        VPC["VPC"]
        IGW["Internet gateway"]
        PUBA["Public subnet A"]
        PUBB["Public subnet B"]
        PRIVA["Private subnet A"]
        PRIVB["Private subnet B"]
        NATA["NAT A"]
        NATB["NAT B"]
        PUBRT["Public route table"]
        PRIVRTA["Private route table A"]
        PRIVRTB["Private route table B"]
    
        VPC --> IGW
        VPC --> PUBA
        VPC --> PUBB
        VPC --> PRIVA
        VPC --> PRIVB
        PUBA --> NATA
        PUBB --> NATB
        PUBRT --> IGW
        PRIVRTA --> NATA
        PRIVRTB --> NATB

A version-pinned, reviewed VPC module can create this repeated graph. You still must understand:

• CIDRs and AZs;
• route-table associations;
• NAT count/cost;
• subnet tags/discovery;
• endpoint behavior;
• outputs consumed by EKS/ALB;
• module upgrade changes.

“Module” means reusable abstraction, not outsourced understanding.

State: the ownership database

Example state relationship:

Terraform address:
      aws_s3_bucket.receipts
    
    AWS object:
      bucket receipt-prod-data-a8f3...

If state is lost, configuration still exists, but Terraform no longer knows it owns that bucket. A new plan may propose another bucket.

If two states own the same object:

    flowchart LR
        STATEA["State A says
bucket policy = private"] --> BUCKET["One real bucket"]
        STATEB["State B says
bucket policy = different"] --> BUCKET
        BUCKET --> FIGHT["Alternating drift and unsafe changes"]

One object should have one desired-state authority.

Remote state and locking

terraform {
      backend "s3" {
        bucket       = "receipt-terraform-state-123456789012"
        key          = "platform/prod/terraform.tfstate"
        region       = "eu-west-2"
        encrypt      = true
        use_lockfile = true
      }
    }

Why remote:

• shared team access;
• centralized permissions/audit;
• recovery/versioning;
• CI can use the same authoritative state.

Why lock:

operator A reads state
    operator B reads same state
    A applies update
    B applies stale understanding
    → lost/conflicting ownership update

S3 lock files are the current backend mechanism to use. Many old tutorials create a DynamoDB lock table; that mechanism is deprecated.

The bootstrap puzzle

The backend bucket cannot be created by a state that needs that bucket before initialization.

    flowchart LR
        BOOT["Small bootstrap root / organization foundation"] --> BUCKET["State bucket
versioned, encrypted, restricted"]
        BUCKET --> ROOT["Real infrastructure roots"]

Keep bootstrap ownership explicit. Protect the state bucket more strongly than ordinary lab data.

State boundaries

Useful starting point:

foundation state
      audit/state/account foundations
    
    network state
      VPC, subnets, routes, endpoints
    
    platform state
      EKS, node groups, ECR, platform IAM
    
    GitOps desired state
      application Deployment, Service, Ingress, values

Split when ownership, permissions, change rate, or blast radius differ. Do not make every resource its own state; cross-state dependencies can become another problem.

Plan before apply

    flowchart LR
        CODE["HCL change"] --> FMT["fmt"]
        FMT --> VALIDATE["validate"]
        VALIDATE --> CHECKS["security/policy checks"]
        CHECKS --> PLAN["plan -out=tfplan"]
        PLAN --> REVIEW["human/policy review"]
        REVIEW --> APPLY["apply reviewed plan"]

Commands:

terraform fmt -check -recursive
    terraform init
    terraform validate
    terraform plan -out=tfplan
    terraform show tfplan
    terraform apply tfplan

The saved plan ties review to the executed actions. Treat the plan file as sensitive.

Read plan symbols

+    create
    ~    update in place
    -    destroy
    -/+  destroy then create replacement
    +/-  create replacement then destroy old

High-attention changes:

• IAM permission expansion;
• public access;
• security-group widening;
• route changes;
• EKS/node replacement;
• bucket/log/data deletion;
• “forces replacement” on stateful resources.

Drift

configuration: security group allows ALB only
    console change: someone adds 0.0.0.0/0
    next plan: Terraform proposes removing the extra rule

Terraform is not malfunctioning. The manual change conflicts with declared state.

If the emergency change should remain, encode and review it. Otherwise, allow the declared configuration to restore the intended state.

Import existing resources

Writing:

resource "aws_s3_bucket" "receipts" {
      bucket = "existing-receipt-bucket"
    }

does not automatically make Terraform safely own the existing bucket. Import the real object into the intended address, describe its settings, and review a no-surprise plan.

Refactor without replacement

Renaming a Terraform address can look like delete/create unless you record the move:

moved {
      from = aws_cloudwatch_log_group.api
      to   = aws_cloudwatch_log_group.receipt_api
    }

Intent:

same AWS object
    new Terraform address

not:

destroy log group
    create empty replacement

Lifecycle rules require reasons

lifecycle {
      prevent_destroy = true
    }

This can protect a critical object from Terraform destruction. It does not stop every AWS principal from deleting it.

Other cautions:

• create_before_destroy may need duplicate names/capacity;
• globally unique names can prevent overlap;
• broad ignore_changes hides drift;
• a lifecycle rule can block an intentional recovery operation.

Document why the rule exists.

Terraform must not own the same Deployment as Argo CD

Bad:

    flowchart LR
        TF["Terraform Kubernetes provider"] --> DEP["Receipt Deployment"]
        ARGO["Argo CD"] --> DEP
        DEP --> FIGHT["Two desired-state authorities"]

Better:

Terraform → durable AWS platform and bootstrap
    Argo CD → application Kubernetes resources

Break it

Never force-unlock until you prove the original writer cannot continue.

Prediction

Two Terraform directories contain identical S3 resource code but use different state backends. Do they manage the same bucket automatically?

Does sensitive = true remove a value from Terraform state?

A plan replaces the EKS cluster. Should you apply because validation passed?

Lab 10 — plan as a teaching artifact

Create a disposable root for an S3 bucket, ECR repository, and log group.

For each plan action, annotate:

What real API/object will change?
    Why does Terraform know the dependency?
    Which state owns it?
    What IAM role performs it?
    Could it contain data?
    How will it be restored or deleted?
    What could it cost?

Then:

1. apply a reviewed saved plan;
2. change a harmless tag manually;
3. run plan and identify drift;
4. restore through configuration;
5. rename one Terraform address with a moved block;
6. prove no resource replacement;
7. destroy only the disposable root;
8. verify the remote object inventory.

Completion criterion:

You can explain a plan in AWS actions, not only Terraform symbols.

Feel the problem first

The dangerous “pipeline”:

developer laptop
    → docker build
    → docker push latest
    → kubectl set image production
    → hope

Questions it cannot answer:

• Which source commit produced the running bytes?
• Were the same bytes tested?
• Who approved production?
• Which identity changed the cluster?
• What is the desired production version?
• Will a manual cluster edit persist?
• Can we reproduce or roll back?

We will create three truths:

    flowchart LR
        SOURCE["Source truth
Git commit"]
        ARTIFACT["Artifact truth
ECR digest"]
        DEPLOY["Deployment truth
GitOps commit/values"]
    
        SOURCE --> ARTIFACT --> DEPLOY

Each truth has a different job.

CI versus CD

Continuous Integration

Proves and packages the source:

checkout
    → lint/test
    → build
    → scan/attest
    → push immutable image

Continuous Delivery/Deployment

Promotes and reconciles an approved artifact:

propose digest change
    → review
    → merge desired state
    → reconcile cluster
    → verify user outcome

CI should not casually become a production cluster administrator.

GitHub Actions without long-lived AWS keys

OIDC mental movie

    sequenceDiagram
        participant J as GitHub Actions job
        participant G as GitHub OIDC provider
        participant S as AWS STS
        participant E as ECR
    
        J->>G: Request identity token
        G-->>J: Signed JWT with repository/job claims
        J->>S: AssumeRoleWithWebIdentity
        S->>S: Verify provider, audience, subject, conditions
        S-->>J: Temporary AWS credentials
        J->>E: Push image

Workflow permission:

permissions:
      contents: read
      id-token: write

id-token: write only lets the job request an OIDC token. AWS role trust and permissions decide AWS access.

Trust exactly the intended job

For a protected GitHub environment production-build, an illustrative traditional subject is:

{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:example-org/receipt-api:environment:production-build"
        }
      }
    }

GitHub's immutable subject format for newly created/opted-in repositories includes owner and repository IDs. Inspect the actual token claims for your repository and use the exact format. Do not copy a wildcard subject from an old blog post.

Protect the environment with:

• allowed deployment branches/tags;
• required reviewers where appropriate;
• restricted secrets/variables;
• no arbitrary forked PR code.

Pull requests are untrusted input

Safe split:

    flowchart TB
        PR["Pull request code"] --> SAFE["Lint, tests, render, build without prod push"]
        MAIN["Protected main/environment"] --> PRIV["OIDC role, publish artifact,
propose promotion"]

Do not give unreviewed PR code production AWS credentials.

Be especially careful with pull_request_target: it runs with privileges from the base repository context. Never casually check out and execute attacker-controlled code there.

Build workflow

name: Build receipt image
    
    on:
      push:
        branches: [main]
    
    permissions:
      contents: read
      id-token: write
    
    env:
      AWS_REGION: eu-west-2
      ECR_REPOSITORY: receipt-api
    
    jobs:
      image:
        runs-on: ubuntu-latest
        environment: production-build
    
        steps:
          - name: Check out source
            uses: actions/checkout@v6
    
          - name: Test
            run: ./scripts/ci-test.sh
    
          - name: Obtain temporary AWS credentials
            uses: aws-actions/configure-aws-credentials@v6
            with:
              role-to-assume: arn:aws:iam::123456789012:role/receipt-github-ecr-publisher
              aws-region: ${{ env.AWS_REGION }}
    
          - name: Log in to ECR
            id: ecr
            uses: aws-actions/amazon-ecr-login@v2
    
          - name: Set up Buildx
            uses: docker/setup-buildx-action@v4
    
          - name: Build and push
            id: build
            uses: docker/build-push-action@v7
            with:
              context: .
              push: true
              tags: |
                ${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:git-${{ github.sha }}
              provenance: true
              sbom: true
    
          - name: Print immutable reference
            run: |
              printf '%s@%s\n' \
                '${{ steps.ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}' \
                '${{ steps.build.outputs.digest }}'

Production practice: pin actions to reviewed commit SHAs and use dependency automation to propose updates. Major tags make this teaching example readable but can move.

The publisher role should push to the intended ECR repository. It does not need Terraform or EKS admin permission.

Helm: templates become concrete Kubernetes YAML

Raw YAML copied per environment drifts:

dev/deployment.yaml
    stage/deployment.yaml
    prod/deployment.yaml

Helm creates a reusable chart:

charts/receipt-api/
    ├── Chart.yaml
    ├── values.yaml
    ├── values.schema.json
    └── templates/
        ├── deployment.yaml
        ├── service.yaml
        ├── ingress.yaml
        ├── serviceaccount.yaml
        └── pdb.yaml

Mental model:

    flowchart LR
        T["Chart templates"]
        D["Default values"]
        P["Production values"]
        H["Helm renderer"]
        Y["Concrete Kubernetes YAML"]
    
        T --> H
        D --> H
        P --> H
        H --> Y

Helm is a package and rendering system. It does not make unsafe configuration safe.

Design a small values interface

replicaCount: 3
    
    image:
      repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api
      digest: sha256:222...
    
    service:
      port: 80
      targetPort: 8080
    
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        memory: 512Mi
    
    ingress:
      host: api.example.com

Good values express intended variability. A value for every line of YAML merely hides the Kubernetes object behind a harder interface.

Template the immutable image

containers:
      - name: api
        image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"
        imagePullPolicy: IfNotPresent

Render before deployment:

helm lint ./charts/receipt-api
    
    helm template receipt-api ./charts/receipt-api \
      --namespace receipt \
      --values ./environments/prod/receipt-api.yaml

The rendered YAML—not the pretty values file—is what Kubernetes will receive.

Validate values

values.schema.json can catch bad inputs:

{
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "type": "object",
      "required": ["replicaCount", "image"],
      "properties": {
        "replicaCount": {
          "type": "integer",
          "minimum": 2
        },
        "image": {
          "type": "object",
          "required": ["repository", "digest"],
          "properties": {
            "repository": {
              "type": "string",
              "minLength": 1
            },
            "digest": {
              "type": "string",
              "pattern": "^sha256:[a-f0-9]{64}$"
            }
          }
        }
      }
    }

Schema validation proves value shape, not runtime health.

Helm values are not a secret store

Bad:

databasePassword: super-secret

committed in a values file.

A Kubernetes Secret is base64-encoded, not encrypted merely because its kind is Secret.

Use an approved pattern:

• External Secrets controller reading AWS Secrets Manager/Parameter Store;
• encrypted Git secret workflow with reviewed keys/controllers;
• organization secret platform.

The controller/workload gets its own IAM permission. Do not print rendered secrets in CI logs.

GitOps repository

One simple shape:

receipt-deploy/
    ├── charts/
    │   └── receipt-api/
    ├── environments/
    │   ├── dev/
    │   │   └── receipt-api.yaml
    │   └── prod/
    │       └── receipt-api.yaml
    └── argocd/
        └── receipt-prod.yaml

Application repository:

source code
    tests
    Dockerfile
    CI workflow

Deployment repository:

chart
    environment values
    approved image digest
    Argo CD application definition

The split is not mandatory, but it makes source build and environment promotion separate review boundaries.

Promotion is a Git diff

CI opens a pull request:

 image:
       repository: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/receipt-api
    -  digest: sha256:111...
    +  digest: sha256:222...

The PR should include:

• source commit;
• image digest;
• test/scan/provenance evidence;
• rendered manifest diff;
• change risk;
• rollback digest.

Merging changes desired state. CI does not need to run kubectl.

Argo CD: compare target and live state

    flowchart LR
        GIT["Git target state"]
        RENDER["Helm renders"]
        DIFF["Compare"]
        LIVE["EKS live state"]
        SYNC["Sync"]
    
        GIT --> RENDER --> DIFF
        LIVE --> DIFF
        DIFF -->|"OutOfSync"| SYNC --> LIVE

Core meanings:

Synced and Healthy are separate.

Argo CD uses Helm as a renderer

When Argo CD manages a Helm source:

Helm renders manifests
    Argo CD owns deployment lifecycle

Do not also run helm upgrade independently for the same application. That creates two owners.

Argo CD Application

apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: receipt-prod
      namespace: argocd
    spec:
      project: receipt
    
      source:
        repoURL: https://github.com/example-org/receipt-deploy.git
        targetRevision: main
        path: charts/receipt-api
        helm:
          valueFiles:
            - ../../environments/prod/receipt-api.yaml
    
      destination:
        server: https://kubernetes.default.svc
        namespace: receipt
    
      syncPolicy:
        automated:
          enabled: true
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Understand the switches:

• automated: reconcile new desired changes automatically;
• prune: delete managed objects removed from desired state;
• selfHeal: restore Git state after live drift;
• allowEmpty: if enabled with pruning, an empty desired application can remove all

managed objects—leave off unless intentional.

An AppProject should restrict source repositories, destination clusters/namespaces, and resource kinds.

Manual edits and self-heal

    sequenceDiagram
        participant O as Operator
        participant K as EKS
        participant A as Argo CD
        participant G as Git
    
        O->>K: kubectl edit replicas 3 → 8
        A->>G: Read target replicas = 3
        A->>K: Observe live replicas = 8
        A->>K: Restore replicas = 3

The platform did what it was told. During an emergency:

1. use a documented break-glass process;
2. pause/narrow reconciliation intentionally if necessary;
3. record the manual action;
4. update Git with the lasting fix;
5. restore normal reconciliation;
6. review why the normal path was too slow.

Rollback

Normal GitOps rollback:

revert digest-change commit
    → review/merge
    → Argo CD renders old digest
    → Kubernetes rolls back workload

It succeeds only if:

• old image still exists;
• config remains compatible;
• data/schema changes are backward-compatible;
• dependencies still support the old version.

Rollback is an application-system property, not merely a UI button.

Bootstrap: who installs Argo CD?

Argo CD cannot deploy into a cluster that does not exist.

    flowchart LR
        TF["Terraform
VPC, EKS, IAM, ECR"] --> BOOT["Controlled one-time bootstrap
install Argo CD + root app"]
        BOOT --> ARGO["Argo CD"]
        ARGO --> ADDONS["Platform add-ons"]
        ARGO --> APPS["Receipt and other apps"]

Terraform or a controlled script may use Helm for the initial Argo CD install. After handoff, document exactly which layer owns:

• Argo CD version/config;
• add-ons;
• application resources.

Avoid cycles:

Terraform waits for an Argo-managed controller
    while Argo waits for a Terraform-managed object

The complete deployment mental movie

    sequenceDiagram
        participant D as Developer
        participant G as GitHub Actions
        participant E as ECR
        participant R as GitOps repo
        participant A as Argo CD
        participant H as Helm renderer
        participant K as EKS
        participant L as ALB
        participant C as CloudWatch
    
        D->>G: Push source commit
        G->>G: Test and build
        G->>E: Push immutable image
        E-->>G: Digest sha256:222...
        G->>R: Open digest-update PR
        R->>R: Review and merge
        A->>R: Detect target change
        A->>H: Render chart + prod values
        H-->>A: Kubernetes manifests
        A->>K: Apply Deployment change
        K->>E: Nodes pull exact digest
        K->>K: Start and probe new pods
        K->>L: Register healthy targets
        L->>K: Send traffic
        K-->>C: Logs and metrics

What each tool owns

This table is the antidote to “just connect all the tools.”

Break it — identify which truth failed

Prediction

Should the build workflow call kubectl apply after pushing ECR?

Argo CD says OutOfSync immediately after an operator changes replicas. Is that an Argo failure?

Helm lint succeeds. Does that prove the pod can call S3?

Lab 11 — promote, observe, revert

1. build and push an image from a protected workflow;
2. record source commit and ECR digest;
3. open a GitOps PR changing only the dev digest;
4. inspect rendered-manifest diff;
5. merge;
6. watch Argo sync;
7. watch Deployment, ReplicaSet, pods, readiness, and ALB target health;
8. place a deployment marker on the dashboard;
9. verify user success;
10. revert the Git commit;
11. watch the previous digest return.

Completion criterion:

No one needs to remember a kubectl command to explain what production should run.

“Production-ready” is not a service

It means we have evidence for properties such as:

available
    secure
    observable
    recoverable
    repeatable
    scalable
    cost-aware
    operable during change

Installing EKS does not prove any of them.

Final runtime trace — one receipt from DNS to durable object

The user sends:

POST https://api.example.com/receipts

Trace 1 — find the endpoint

browser
    → recursive DNS resolver
    → Route 53 authoritative hosted zone
    → api.example.com alias
    → ALB addresses

Evidence:

dig api.example.com
    dig NS example.com

Trace 2 — enter the VPC-facing service

browser opens TLS to ALB:443
    → ALB security group permits connection
    → listener presents certificate
    → host/path rule selects Receipt target group

Evidence:

curl --verbose https://api.example.com/health/ready

Trace 3 — select healthy code

target group contains healthy pod IPs
    → ALB selects 10.20.10.31:8080
    → app security group accepts source ALB SG
    → Receipt process receives request

Evidence:

aws elbv2 describe-target-health --target-group-arn "$TARGET_GROUP_ARN"
    kubectl -n receipt get pods,endpointslices -o wide

Trace 4 — authorize the human

Receipt API validates the application's user/session and the upload. This is application authentication, not AWS IAM.

Evidence:

request ID
    controlled 401/403/4xx response
    application audit/business event

Trace 5 — obtain workload identity

pod uses ServiceAccount receipt-api
    → EKS Pod Identity association selects receipt-prod-s3-writer
    → SDK obtains temporary credentials

Evidence:

kubectl -n receipt get pod "$POD" \
      -o jsonpath='{.spec.serviceAccountName}{"\n"}'
    
    aws eks list-pod-identity-associations \
      --cluster-name receipt-prod

Never print the credentials themselves.

Trace 6 — reach and authorize S3

SDK resolves S3 endpoint
    → route uses S3 endpoint or designed NAT/public-service path
    → TLS reaches S3
    → S3 evaluates role, bucket, endpoint, organization, and KMS controls
    → object version is stored

Evidence:

application dependency log
    AWS request ID
    S3 object/version metadata
    CloudTrail data event when configured

Trace 7 — explain the result

Receipt API returns 201
    → structured log contains request ID, digest, duration, result
    → custom/business metric records durable receipt
    → ALB metrics record status and target time

The whole mental movie:

    sequenceDiagram
        participant U as User
        participant D as Route 53
        participant A as ALB
        participant P as Receipt pod
        participant I as Pod Identity
        participant S as S3
        participant C as CloudWatch
    
        U->>D: Resolve api.example.com
        D-->>U: ALB addresses
        U->>A: HTTPS POST /receipts
        A->>P: HTTP to healthy pod IP
        P->>P: Authenticate user and validate PDF
        P->>I: Request temporary AWS credentials
        I-->>P: Scoped role session
        P->>S: Signed PutObject
        S-->>P: Object stored
        P-->>A: 201 Created
        A-->>U: 201 Created
        P-->>C: Structured result + metrics

Final change trace — one commit becomes healthy traffic

1. developer pushes source commit
    2. GitHub Actions tests
    3. protected job obtains AWS role through OIDC
    4. build produces image
    5. image is pushed to ECR
    6. ECR returns digest
    7. automation opens GitOps digest PR
    8. review merges desired change
    9. Argo CD detects target-state change
    10. Helm renders chart with production values
    11. Argo applies Deployment change
    12. Deployment creates new ReplicaSet
    13. scheduler assigns pods
    14. node pulls ECR digest
    15. startup probe passes
    16. readiness probe passes
    17. Service endpoints include pods
    18. ALB targets become healthy
    19. old pods drain
    20. SLO metrics verify outcome

    flowchart LR
        CODE["Source commit"] --> TEST["CI proves"]
        TEST --> ECR["ECR digest"]
        ECR --> PR["GitOps PR"]
        PR --> ARGO["Argo CD"]
        ARGO --> HELM["Helm render"]
        HELM --> EKS["EKS rollout"]
        EKS --> ALB["Healthy ALB targets"]
        ALB --> SLO["User outcome"]

Deployment is not finished at “pod started.” It is finished when intended user behavior is healthy.

Availability: count failure domains, not logos

Desired shape:

    flowchart TB
        ALB["ALB across AZ A + AZ B"]
        subgraph A["AZ A"]
            NA["Node A"]
            PA["Receipt pod A"]
            NATA["NAT/required endpoints A"]
        end
        subgraph B["AZ B"]
            NB["Node B"]
            PB["Receipt pod B"]
            NATB["NAT/required endpoints B"]
        end
    
        ALB --> PA
        ALB --> PB
        NA --> PA
        NB --> PB

Ask:

• are replicas actually spread?
• does each AZ have worker capacity?
• does each private subnet have required egress/endpoints?
• does a shared dependency reintroduce one-AZ failure?
• can remaining capacity handle traffic when one AZ is removed?

Two pods in one AZ are replication, not AZ resilience.

Define success with SLI and SLO

SLI

Measured behavior:

successful durable receipt writes / valid receipt requests

SLO

Target:

99.9% successful over the chosen rolling window

Error budget

The tolerated failure implied by the SLO.

SLOs turn arguments into decisions:

• how many replicas/AZs?
• how quickly page?
• how risky can a rollout be?
• should feature work pause while reliability improves?

RTO and RPO

“Versioning enabled” does not prove an RPO. Run restore exercises and measure.

Rollouts need spare capacity

strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 0
        maxSurge: 1

This asks for one extra pod while retaining all desired available replicas.

It requires spare:

• node CPU/memory;
• subnet IP;
• ECR pull/network capacity;
• quota;
• dependency capacity.

A safe-looking rollout setting without surge capacity can stall.

Old and new versions coexist

During rollout:

    flowchart LR
        USER["Requests"] --> ALB["ALB"]
        ALB --> OLD["v1 pods"]
        ALB --> NEW["v2 pods"]
        OLD --> DATA["Shared data/dependencies"]
        NEW --> DATA

Therefore:

• APIs/messages need compatible schemas;
• database changes should expand, migrate, then contract later;
• config should work during overlap;
• old version must tolerate data written by new version if rollback is possible.

Kubernetes schedules coexistence. It cannot invent compatibility.

Security review by boundary

Identity

• federated human roles and MFA;
• no routine root use;
• distinct Terraform, CI, node, controller, and pod roles;
• narrow trust conditions;
• temporary credentials;
• EKS access entries and RBAC;
• CloudTrail audit.

Network

• public ALB, private worker/application subnets where appropriate;
• ALB SG → app SG reference;
• intentional egress and endpoints;
• restricted EKS API endpoint design;
• NetworkPolicy for pod east-west flows;
• no accidental 0.0.0.0/0 application port.

Workload

• exact image digest;
• reviewed base image and patch cadence;
• vulnerability scanning;
• non-root/read-only filesystem where supported;
• minimal Linux capabilities;
• resource requests/limits;
• admission/policy checks;
• secrets outside images/Git.

Data

• S3 Block Public Access;
• minimum object-prefix permission;
• encryption and KMS policy when used;
• versioning/lifecycle/retention;
• recovery testing;
• classified log contents;
• audit events according to risk.

Supply chain

• protected branches and environments;
• OIDC instead of permanent cloud keys;
• pinned actions/modules/charts;
• artifact digest and provenance;
• build role cannot administer cluster;
• promotion approval is visible in Git.

Observability review

    flowchart TB
        USER["User SLI/SLO"]
        EDGE["Route 53 + ALB"]
        ORCH["EKS + controllers"]
        APP["Receipt application"]
        DEP["S3/AWS dependencies"]
        INFRA["Nodes, network, IP capacity"]
        CHANGE["GitHub + Argo + Terraform change markers"]
    
        USER --> EDGE --> ORCH --> APP --> DEP --> INFRA
        CHANGE -.-> USER

Minimum questions:

• what is the user success rate?
• which digest receives traffic?
• how many healthy targets?
• which pods restart or remain Pending?
• how long do S3 calls take?
• when did a deployment/infra change happen?
• are logs being delivered?
• does every page have an owner/runbook?

Cost review

Look at each design decision:

Cost and resilience trade off. One NAT can be cheaper and a single-AZ dependency. One ALB per tiny service can simplify ownership and increase baseline cost. Make the trade-off explicit.

Recovery inventory

Terraform can recreate a bucket. It cannot reconstruct lost receipt bytes.

Incident 1 — ALB returns 503

    flowchart TD
        S["ALB returns 503"] --> H{"Healthy targets?"}
        H -->|"none"| REG{"Targets registered?"}
        REG -->|"none"| K["Check Ingress controller,
Service endpoints, pods"]
        REG -->|"yes, unhealthy"| HC["Check health reason,
path, port, SG, readiness"]
        H -->|"healthy targets exist"| RULE["Check listener action/rule
and app response source"]

What already worked:

DNS likely resolved
    connection reached ALB
    listener produced a response

Do not start by replacing Route 53 or the whole VPC.

Incident 2 — S3 AccessDenied

1. exact action?
    2. exact bucket/key ARN?
    3. actual caller role session?
    4. correct ServiceAccount/Pod Identity?
    5. role permission?
    6. bucket policy?
    7. endpoint policy?
    8. organization/boundary/session explicit deny?
    9. KMS key permission?
    10. CloudTrail evidence?

Do not fix by granting s3: on .

Incident 3 — pods Pending during rollout

    flowchart TD
        P["Pod Pending"] --> EVT["Read scheduler event"]
        EVT --> CPU{"Insufficient CPU/memory?"}
        CPU -->|"yes"| CAP["Requests, node capacity,
autoscaler"]
        CPU -->|"no"| CON{"Taint/affinity/topology?"}
        CON -->|"yes"| SCHED["Fix intended scheduling contract"]
        CON -->|"no"| IP{"Failed pod IP/ENI?"}
        IP -->|"yes"| CNI["Subnet free IPs,
CNI/node limits"]
        IP -->|"no"| Q["Quota/instance availability/
other event reason"]

Do not blindly reduce requests. A pod that fits the scheduler but OOMs at runtime is not fixed.

Incident 4 — Argo CD OutOfSync

Read the diff:

• human live edit?
• another controller defaults/owns a field?
• non-deterministic Helm output?
• admission/RBAC apply failure?
• source cannot render?
• object should be excluded from comparison?

Clicking Sync repeatedly is not diagnosis.

Incident 5 — Terraform state lock

Is another run active?
    → yes: wait/coordinate
    → no: prove the writer cannot resume
    → only then use approved force-unlock procedure

Two concurrent writers are worse than a stale lock.

Production readiness questions

Answer these in writing:

1. Which account, Region, VPC, and AZs?
2. Which resources are public?
3. Which role does every human, CI job, controller, node, and pod use?
4. Which exact image digest is deployed?
5. Which Git commit says production should run it?
6. Who owns VPC, EKS, Ingress, ALB, and DNS desired state?
7. What is the user SLO?
8. Which alarms page a person and which create tickets?
9. How does one-AZ loss affect traffic and egress?
10. How do receipt objects recover?
11. How does a bad deployment revert?
12. Which schema/data changes prevent rollback?
13. Which capacity limits block surge?
14. What is the monthly cost baseline?
15. How are EKS, add-on, provider, action, and image versions upgraded?
16. What happens when GitHub, ECR, or Argo CD is temporarily unavailable?
17. Where is the break-glass procedure?
18. Has teardown/recovery been rehearsed?

“The platform handles it” is not an answer.

Capstone lab — prove the whole system

Phase A: platform

Terraform creates:

• VPC across two AZs;
• intended public/private subnets and routes;
• endpoints/NAT according to design;
• ECR repository;
• private versioned S3 bucket;
• IAM roles;
• CloudWatch log groups;
• EKS and managed nodes;
• Route 53/ALB foundations according to ownership.

Evidence:

reviewed plan
    state location and lock
    resource tags
    account/Region proof
    cost estimate

Phase B: application

GitHub Actions:

• tests;
• assumes ECR role through OIDC;
• builds once;
• pushes;
• records digest;
• opens GitOps PR.

Evidence:

source commit → image digest
    temporary role session
    test/scan/provenance result

Phase C: deployment

Argo CD:

• renders Helm chart;
• syncs EKS;
• Deployment rolls out;
• probes pass;
• ALB target becomes healthy.

Evidence:

GitOps commit → rendered image digest
    Argo sync/health
    pod imageID
    target health

Phase D: runtime

Send a receipt:

• Route 53 resolves;
• ALB routes;
• pod authorizes;
• Pod Identity obtains credentials;
• S3 stores object;
• logs/metrics show request;
• 201 reaches user.

Evidence:

one request ID across app evidence
    object key/version
    deployed digest
    latency/error metrics

Phase E: failure

Break one item at a time:

• wrong Service selector;
• wrong readiness path;
• missing S3 permission;
• pod request too large;
• live replica edit with self-heal;
• stale Terraform lock simulation.

For every failure:

prediction
    observable symptom
    first useful evidence
    failed contract
    desired-state repair
    verification
    runbook/alert improvement

Completion criterion:

You can narrate both the successful path and the broken path without saying “AWS somehow routes it.”

# Final Knowledge Check

Try each answer aloud before opening it.

1. IAM versus security group

The pod can connect to the S3 endpoint but gets AccessDenied. Should you open port 443 wider?

2. Public subnet

What makes a subnet public?

3. EC2 health

Why is running weaker than application readiness?

4. S3 key

Why is S3 not just “a folder in the cloud”?

5. ECR identity

Which value proves exact image content?

6. CloudWatch

What four operational signals should every request service cover?

7. Route 53 versus ALB

Which service chooses a healthy Receipt pod for an HTTP request?

8. ECS roles

ECR pull works; S3 write fails. Which ECS role is the first suspect?

9. EKS identities

Why can a developer authenticate to EKS yet be forbidden from listing pods?

10. Kubernetes Service

A Service exists but has zero endpoints. What likely relationship failed?

11. Terraform state

Why is configuration alone insufficient for an existing resource?

12. Helm and Argo CD

Who renders, and who reconciles?

13. CI/CD

Why does CI update a GitOps digest instead of running kubectl?

14. Availability

Why are three pods not automatically highly available?

15. Production

When is a deployment complete?

# A-to-Z Memory Map

# The One-Page Mental Model

WHO MAY ACT?
      IAM roles, trust, temporary credentials, policy
    
    WHERE MAY PACKETS TRAVEL?
      VPC, subnet, route, gateway/endpoint, SG/NACL, DNS
    
    WHERE DOES CODE RUN?
      EC2 machine
      ECS task/service
      EKS pod/Deployment
    
    WHERE DO BYTES LIVE?
      EBS = block
      S3 = object
      ECR = container artifact
    
    HOW DO USERS FIND HEALTHY CODE?
      Route 53 name → ALB listener/rule → healthy target
    
    HOW DO WE KNOW?
      CloudWatch metrics/logs/alarms + application evidence
      CloudTrail for AWS API activity
    
    HOW IS THE PLATFORM REPRODUCED?
      Terraform config + state → reviewed plan → apply
    
    HOW DOES SOFTWARE MOVE?
      source commit → CI → ECR digest → GitOps change
      → Argo CD → Helm render → EKS rollout → SLO verification
    
    WHO OWNS EACH OBJECT?
      one desired-state authority per resource/field

For any failure, ask:

1. Name: did it resolve?
    2. Route: can packets reach and return?
    3. Filter: do network controls permit the flow?
    4. Identity: which credentials are actually used?
    5. Authorization: is the exact action/resource allowed?
    6. Health: is the target ready, not merely alive?
    7. Ownership: which controller should make this state true?
    8. Evidence: what proves every answer?

If you can use those eight questions, the platform is no longer a collection of logos. It is a system you can reason about.

# Official Documentation Map

Use these primary sources to check current behavior:

• [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
• [IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html)
• [GitHub OIDC roles in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html)
• [VPC/subnet basics](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-subnet-basics.html)
• [VPC route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html)
• [VPC security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html)
• [NAT devices](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.html)
• [AWS PrivateLink/VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/index.html)
• [EC2 lifecycle](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html)
• [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)
• [S3 concepts](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
• [ECR concepts](https://docs.aws.amazon.com/AmazonECR/latest/userguide/concept-and-components.html)
• [CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html)
• [CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html)
• [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)
• [ALB target health](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html)
• [Route 53 alias records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html)
• [ECS task definitions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html)
• [ECS task roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)
• [EKS architecture](https://docs.aws.amazon.com/eks/latest/userguide/eks-architecture.html)
• [EKS access control](https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html)
• [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)
• [EKS VPC CNI](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html)
• [AWS Load Balancer Controller on EKS](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html)
• [Terraform S3 backend/locking](https://developer.hashicorp.com/terraform/language/backend/s3)
• [Terraform provider requirements](https://developer.hashicorp.com/terraform/language/providers/requirements)
• [Terraform AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest)
• [GitHub Actions OIDC with AWS](https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws)
• [GitHub OIDC claims](https://docs.github.com/en/actions/reference/security/oidc)
• [Helm charts](https://helm.sh/docs/topics/charts/)
• [Helm values](https://helm.sh/docs/chart_template_guide/values_files/)
• [Argo CD](https://argo-cd.readthedocs.io/en/stable/)
• [Argo CD automated sync](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/)
• [Argo CD with Helm](https://argo-cd.readthedocs.io/en/latest/user-guide/helm/)
• [Kubernetes disruptions/PDBs](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)

Check official documentation before choosing production versions, quotas, security settings, or architecture.

Teaching edition reviewed against current official documentation on 23 July 2026.

Try each answer aloud before opening it.

Try each answer aloud before opening it.

1. IAM versus security group

The pod can connect to the S3 endpoint but gets AccessDenied. Should you open port 443 wider?

2. Public subnet

What makes a subnet public?

3. EC2 health

Why is running weaker than application readiness?

4. S3 key

Why is S3 not just “a folder in the cloud”?

5. ECR identity

Which value proves exact image content?

6. CloudWatch

What four operational signals should every request service cover?

7. Route 53 versus ALB

Which service chooses a healthy Receipt pod for an HTTP request?

8. ECS roles

ECR pull works; S3 write fails. Which ECS role is the first suspect?

9. EKS identities

Why can a developer authenticate to EKS yet be forbidden from listing pods?

10. Kubernetes Service

A Service exists but has zero endpoints. What likely relationship failed?

11. Terraform state

Why is configuration alone insufficient for an existing resource?

12. Helm and Argo CD

Who renders, and who reconciles?

13. CI/CD

Why does CI update a GitOps digest instead of running kubectl?

14. Availability

Why are three pods not automatically highly available?

15. Production

When is a deployment complete?

WHO MAY ACT?
  IAM roles, trust, temporary credentials, policy

WHERE MAY PACKETS TRAVEL?
  VPC, subnet, route, gateway/endpoint, SG/NACL, DNS

WHERE DOES CODE RUN?
  EC2 machine
  ECS task/service
  EKS pod/Deployment

WHERE DO BYTES LIVE?
  EBS = block
  S3 = object
  ECR = container artifact

HOW DO USERS FIND HEALTHY CODE?
  Route 53 name → ALB listener/rule → healthy target

HOW DO WE KNOW?
  CloudWatch metrics/logs/alarms + application evidence
  CloudTrail for AWS API activity

HOW IS THE PLATFORM REPRODUCED?
  Terraform config + state → reviewed plan → apply

HOW DOES SOFTWARE MOVE?
  source commit → CI → ECR digest → GitOps change
  → Argo CD → Helm render → EKS rollout → SLO verification

WHO OWNS EACH OBJECT?
  one desired-state authority per resource/field

For any failure, ask:

1. Name: did it resolve?
2. Route: can packets reach and return?
3. Filter: do network controls permit the flow?
4. Identity: which credentials are actually used?
5. Authorization: is the exact action/resource allowed?
6. Health: is the target ready, not merely alive?
7. Ownership: which controller should make this state true?
8. Evidence: what proves every answer?

If you can use those eight questions, the platform is no longer a collection of logos. It is a system you can reason about.

Use these primary sources to check current behavior:

• [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
• [IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html)
• [GitHub OIDC roles in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html)
• [VPC/subnet basics](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-subnet-basics.html)
• [VPC route tables](https://docs.aws.amazon.com/vpc/latest/userguide/subnet-route-tables.html)
• [VPC security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html)
• [NAT devices](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.html)
• [AWS PrivateLink/VPC endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/index.html)
• [EC2 lifecycle](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html)
• [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)
• [S3 concepts](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
• [ECR concepts](https://docs.aws.amazon.com/AmazonECR/latest/userguide/concept-and-components.html)
• [CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html)
• [CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html)
• [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html)
• [ALB target health](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html)
• [Route 53 alias records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html)
• [ECS task definitions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html)
• [ECS task roles](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)
• [EKS architecture](https://docs.aws.amazon.com/eks/latest/userguide/eks-architecture.html)
• [EKS access control](https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html)
• [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)
• [EKS VPC CNI](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html)
• [AWS Load Balancer Controller on EKS](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html)
• [Terraform S3 backend/locking](https://developer.hashicorp.com/terraform/language/backend/s3)
• [Terraform provider requirements](https://developer.hashicorp.com/terraform/language/providers/requirements)
• [Terraform AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest)
• [GitHub Actions OIDC with AWS](https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws)
• [GitHub OIDC claims](https://docs.github.com/en/actions/reference/security/oidc)
• [Helm charts](https://helm.sh/docs/topics/charts/)
• [Helm values](https://helm.sh/docs/chart_template_guide/values_files/)
• [Argo CD](https://argo-cd.readthedocs.io/en/stable/)
• [Argo CD automated sync](https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/)
• [Argo CD with Helm](https://argo-cd.readthedocs.io/en/latest/user-guide/helm/)
• [Kubernetes disruptions/PDBs](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)

Check official documentation before choosing production versions, quotas, security settings, or architecture.

Teaching edition reviewed against current official documentation on 23 July 2026.