Learn Kubernetes
From First Principles to Deep Dive

Before touching a single YAML file, understand the problem. Kubernetes is an answer. First, let's find the question.

The story in three acts

flowchart LR
    subgraph A["Act 1 · ~2005 · The Server"]
        A1["One machine\nOne app\nDeploy = ssh + copy files\nScale = buy a bigger machine"]
    end
    subgraph B["Act 2 · ~2010 · Virtual Machines"]
        B1["Many VMs per machine\nMany apps\nDeploy = provision VMs (slow)\nScale = more VMs (heavy)"]
    end
    subgraph C["Act 3 · ~2015+ · Containers Everywhere"]
        C1["Hundreds of microservices\nHundreds of machines\nThousands of containers\n\nWHO IS MANAGING ALL THIS?"]
    end
    A --> B --> C
    

Containers solved packaging ("it works on my machine"). But they created a new problem: you now have thousands of moving parts spread over hundreds of machines, and no one driving.

The five pains of containers at scale

Google had already solved this internally with a system called Borg. Kubernetes (Greek for helmsman; "K8s" = K + 8 letters + s) is that design, open-sourced in 2014, built on four ideas. These four ideas are the whole course.

First Principle #1 — Declare the destination, not the route

flowchart TD
    subgraph IMP["IMPERATIVE — you describe HOW (scripts)"]
        I1["ssh server-7, start container"] --> I2["check it stayed up"]
        I2 --> I3["if dead: ssh back in, restart"]
        I3 --> I4["write more scripts for every\nfailure mode you can imagine..."]
        I4 -.->|"3am: a failure you didn't script"| I5["OUTAGE"]
    end
    subgraph DEC["DECLARATIVE — you describe WHAT (desired state)"]
        D1["'Run 3 replicas of app:v2.\nAlways. I don't care where.'"]
        D1 --> D2["The system perpetually works\nto make that statement true"]
    end
    

Imperative: you give step-by-step instructions. Every failure mode is your problem. Declarative: you state the desired end state. Making it true is the system's problem.

Why it wins:

• Idempotent — apply the same declaration twice, get the same result.
• Recoverable — if the system crashes, it reboots, re-reads the desired state, and resumes.
• Auditable — the desired state is a file. It can live in Git. You can diff reality against intent.

First Principle #2 — Reconciliation loops (control theory)

Who makes the desired state true? Not a one-shot installer — a loop that runs forever. The perfect analogy is a thermostat: you set 22°C, and it endlessly measures actual temp → compares → acts → repeats.

flowchart TD
    DESIRED["DESIRED STATE\n'3 replicas of app:v2'\n(written by you)"] --> LOOP
    subgraph LOOP["Controller — runs forever"]
        O["1. OBSERVE\nread actual state"] --> C["2. COMPARE\ndiff actual vs desired"]
        C --> A["3. ACT\ncreate / delete / replace\nto close the gap"]
        A --> O
    end
    LOOP -->|"actions change the world"| ACTUAL["ACTUAL STATE\n2 replicas running"]
    ACTUAL --> LOOP
    

Kubernetes is not a deployment tool. It is a fleet of thermostats.

Deep dive: level-based, not edge-triggered

Naive automation reacts to events: "pod died → start a pod." If you miss the event, the system is silently wrong forever. Kubernetes controllers are level-based: they don't react to the edge of an event; they continuously re-examine the entire current state against the desired state. Any component can crash at any time, and the system converges back to correct.

First Principle #3 — One front door, one source of truth

1. One source of truth — a database called etcd stores the entire desired and observed state.
2. One front door — the API server is the only component allowed to talk to etcd. Everyone else talks only to the API server. Components never call each other directly; they watch the API server for changes.

flowchart TD
    U["You (kubectl)"] --> API
    CTL["Controllers"] --> API
    SCH["Scheduler"] --> API
    K1["Kubelet · node 1"] --> API
    K2["Kubelet · node 2"] --> API
    K3["Kubelet · node 3"] --> API
    API["kube-apiserver\nTHE FRONT DOOR\nauthentication · authorization · validation"]
    API --> ETCD[("etcd\nthe single source of truth")]
    

Think bulletin board, not phone tree. Nobody calls anybody. Consequences:

• Any component can die and restart without losing sync — it just re-reads the board.
• Components are loosely coupled — swap the scheduler for a custom one; nothing else changes.
• One choke point = one place to enforce security and validation on every change.

First Principle #4 — Spec vs Status: the gap is the work

Every object in Kubernetes has two halves:

spec:        # ← YOU write this: the DESIRED state
  replicas: 3

status:      # ← KUBERNETES writes this: the OBSERVED state
  replicas: 2
  conditions:
    - type: Available
      status: "False"

The gap between spec and status is the controller's to-do list. When they match, the system is healthy and the loops idle.

flowchart TD
    A["You run:\nkubectl scale deployment nginx --replicas=5"]
    B["Kubernetes API stores desired state\n\nspec.replicas = 5"]
    C["Deployment Controller checks the object"]
    D["Observed state\n\nstatus.readyReplicas = 2"]
    E{"Does desired state\nmatch observed state?"}
    F["No\n\nDesired: 5\nRunning: 2\nGap: 3 Pods"]
    G["Deployment Controller tells the\nReplicaSet to create 3 Pods"]
    H["ReplicaSet creates\nPod 3, Pod 4, and Pod 5"]
    I["Scheduler assigns each Pod\nto a worker node"]
    J["Kubelet starts the containers"]
    K["Kubernetes updates status\n\nstatus.readyReplicas = 5"]
    L{"Does desired state\nmatch observed state?"}
    M["Yes\n\nspec.replicas = 5\nstatus.readyReplicas = 5\n\nNo more work needed"]
    A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K --> L
    L -->|Yes| M
    L -->|No, some Pods are not ready| C
    

Recap — the whole course in one paragraph

Kubernetes exists because running thousands of containers across hundreds of machines by hand is impossible. Its answer: you declare desired state (Principle 1) as objects with a spec; a set of never-ending reconciliation loops (Principle 2) drive status toward spec (Principle 4); and everything coordinates through a single API server backed by etcd (Principle 3) so the system is loosely coupled, self-healing, and convergent.

Kubernetes orchestrates containers. To understand the orchestrator from first principles, you must first understand the primitive it orchestrates.

What a container REALLY is

A container is not a lightweight virtual machine. A container is an ordinary Linux process that has been given three things by the kernel:

flowchart TD
    P["A normal Linux process\n(your app)"]
    P --> N["1 · NAMESPACES\ncontrol what the process can SEE\nits own: process tree, network, filesystem,\nhostname, users"]
    P --> C["2 · CGROUPS\ncontrol what the process can USE\nmax CPU, max memory, I/O"]
    P --> I["3 · IMAGE (union filesystem)\ncontrols what the process HAS\nlayered, immutable filesystem\nwith its libs and deps baked in"]
    

There is no "container" object in the kernel. There are just processes wearing namespace blinders and cgroup rations.

Containers vs Virtual Machines

Why containers alone are not enough

Docker solves: "package my app and run it anywhere." It does not solve placement, healing, scaling, discovery, or rollouts. Containers created the scale; scale created the pains; the pains demanded an orchestrator.

Quick facts you'll need later

• A container runtime (containerd, CRI-O) actually creates containers via the CRI (Container Runtime Interface). Docker daemon is no longer used directly by Kubernetes.
• Image: immutable, named, versioned (nginx:1.27). Pulled from registries.
• One process per container is the convention — the container's lifetime is tied to that process. This is why the Pod exists.

A Kubernetes cluster has exactly two kinds of machines: the control plane (the brain, which decides) and worker nodes (the muscle, which does).

The map

flowchart TD
    USER["You — kubectl / CI pipeline"] --> API
    subgraph CP["CONTROL PLANE — the brain (decides, remembers)"]
        API["kube-apiserver\nTHE FRONT DOOR\nthe only component that talks to etcd"]
        ETCD[("etcd\nSOURCE OF TRUTH\nall desired + observed state")]
        SCHED["kube-scheduler\ndecides WHERE each pod runs"]
        CM["kube-controller-manager\nruns the reconciliation loops"]
        CCM["cloud-controller-manager\nglues cloud LB/disk/node APIs"]
        API --- ETCD
        SCHED --> API
        CM --> API
        CCM --> API
    end
    subgraph W1["WORKER NODE 1 — the muscle"]
        K1["kubelet\nnode captain"] --> API
        KP1["kube-proxy\nnetwork rules"] --> API
        RT1["container runtime\n(containerd)"]
        K1 --> RT1
        POD1["Pods:\nyour containers"]
        RT1 --- POD1
    end
    subgraph W2["WORKER NODE 2"]
        K2["kubelet"] --> API
        KP2["kube-proxy"] --> API
        RT2["container runtime"]
        K2 --> RT2
        POD2["Pods"]
        RT2 --- POD2
    end
    

Control plane components

kube-apiserver — the front door

Stateless, horizontally scalable. Every request walks: authenticate → authorize → admission control → persist to etcd.

sequenceDiagram
    participant C as Client
    participant A as API Server
    participant E as etcd
    C->>A: HTTPS request
    A->>A: 1 · Authenticate
    A->>A: 2 · Authorize (RBAC)
    A->>A: 3 · Admission control — mutate & validate
    A->>E: persist the object
    E-->>A: ok
    A-->>C: response
    

etcd — the source of truth

A distributed key-value store using the Raft consensus algorithm. A write succeeds only when a quorum confirms it. Always deploy an odd number of members.

kube-scheduler — the matchmaker

Watches for Pods with no node assigned, picks the best node (filter → score), and writes the decision back. That's all it does.

kube-controller-manager — the loop runner

One binary running dozens of independent reconciliation loops:

Node components

kubelet — the node captain

Watches for pods assigned to its node, instructs the container runtime via CRI, runs probes, reports status. If a node dies, recovery is the controllers' job, not the kubelet's.

kube-proxy — the traffic cop

Watches Services and programs iptables/IPVS rules so a Service's virtual IP gets translated to a real pod IP.

container runtime — the hands

containerd / CRI-O. Pulls images, assembles namespaces + cgroups, starts containers under kubelet's direction.

Essential add-ons

Almost all control-plane components run as regular pods in kube-system:

kubectl get pods -n kube-system

Kubernetes does not schedule containers. It schedules Pods. Understanding why is the key to the whole object model.

Why not just containers?

A container is tied to one process. But real workloads sometimes need multiple processes that must run on the same machine, talk over localhost, and share files. So Kubernetes invented the Pod: one or more containers that share a network namespace and storage volumes.

flowchart TD
    subgraph POD["POD — 10.244.1.7 (ONE IP for the whole pod)"]
        subgraph NS["Shared network namespace"]
            C1["app container\nlocalhost:8080"]
            C2["log-shipper sidecar\nreads shared logs"]
        end
        V[("shared volume\nemptyDir")]
        C1 --- V
        C2 --- V
        PAUSE["pause container\n(tiny; just holds the namespaces open)"]
    end
    

Inside a pod: Network — all containers share one IP; Storage — volumes can be mounted by any container; Lifetime — created together, scheduled together, die together.

Rule of thumb: one container per pod, unless the containers are inseparable (share fate, localhost, or files).

Pods are cattle, not pets

Pods are mortal and disposable. If one dies, a controller replaces it. Never SSH into a pod to "fix" it.

Pod lifecycle

stateDiagram-v2
    [*] --> Pending
    Pending --> Running : scheduled, images pulled, containers started
    Running --> Succeeded : all containers exited 0
    Running --> Failed : non-zero exit or pod evicted
    Pending --> Failed : can't schedule / image pull fails
    Running --> Unknown : node stopped reporting
    Succeeded --> [*]
    Failed --> [*]
    

Health probes

Not ready ≠ dead. A pod warming its cache should leave the load balancer without being killed.

Multi-container patterns

flowchart LR
    subgraph SIDECAR["SIDECAR — extends the app"]
        A1["app"] --- S1["log shipper /\nservice-mesh proxy"]
    end
    subgraph AMBASSADOR["AMBASSADOR — speaks for the app"]
        A2["app\n(thinks DB is localhost)"] --- S2["proxy to the\nreal remote DB"]
    end
    subgraph ADAPTER["ADAPTER — translates the app"]
        A3["app\n(custom logs)"] --- S3["reformat logs to a\nstandard for monitoring"]
    end
    

Also: initContainers run before the app containers, in order — perfect for migrations or pre-seeding files.

A minimal Pod, in YAML

apiVersion: v1
kind: Pod
metadata:
  name: web
  labels:
    app: web
spec:
  containers:
    - name: app
      image: nginx:1.27
      ports:
        - containerPort: 80
      readinessProbe:
        httpGet: { path: /, port: 80 }

A bare pod has no guardian. If its node dies, it is gone forever. This lesson is where the reconciliation loops become real.

ReplicaSet — the "keep N alive" loop

A ReplicaSet is a controller with one job: make the number of running pods matching a label selector equal the desired count. Forever.

flowchart TD
    SPEC["spec.replicas: 3\nselector: app=web"] --> LOOP
    subgraph LOOP["ReplicaSet controller — reconciliation loop"]
        O["1. Observe: list pods\nwith label app=web"] --> C{"2. Compare:\nactual vs 3?"}
        C -->|"actual = 2"| A["3. Create 1 pod\nfrom the pod template"]
        C -->|"actual = 4"| A2["3. Delete 1 pod"]
        C -->|"actual = 3"| OK["Do nothing"]
        A --> O
        A2 --> O
        OK --> O
    end
    

This is self-healing: delete a pod — the loop recreates it. A node dies taking 2 pods with it — the loop creates 2 more on healthy nodes. You declared 3; the universe is bent toward 3.

Deployment — the rollout manager

A Deployment manages ReplicaSets to perform safe updates. Each pod-template change creates a new ReplicaSet; old ones are kept (scaled to 0) as snapshots. Rollback = reactivate an old ReplicaSet.

flowchart LR
    T0["t0\nv1 v1 v1"] --> T1["t1\nv1 v1 v2\n(surge: add one v2)"]
    T1 --> T2["t2\nv1 v2 v2\n(retire one v1)"]
    T2 --> T3["t3\nv2 v2 v2\n(old RS scaled to 0)"]
    

• maxSurge — how many extra pods may exist above the desired count during the swap.
• maxUnavailable — how many pods may be down during the swap.
• New pods only count once Ready (readiness probe makes rollouts safe: v1 isn't retired until v2 can actually serve).

kubectl rollout undo deployment/web     # back to previous ReplicaSet
kubectl rollout history deployment/web  # see revisions

Watch self-healing happen

sequenceDiagram
    participant N as Node 2 (runs one web pod)
    participant K2 as kubelet on Node 2
    participant A as API Server / etcd
    participant NC as Node controller
    participant RS as ReplicaSet controller
    participant S as Scheduler
    participant K1 as kubelet on Node 1 (healthy)
    K2->>A: heartbeats every few seconds
    Note over N,K2: NODE 2 DIES (power loss)
    K2 -x A: heartbeats stop
    NC->>A: mark Node 2 NotReady (~40s of silence)
    NC->>A: evict its pods (~5 min toleration expires)
    Note over RS: loop pass: sees 2 of 3 pods
    RS->>A: create replacement pod
    S->>A: bind pod → Node 1
    K1->>K1: pull image, start container
    K1->>A: status: Running + Ready
    Note over RS: loop pass: sees 3 of 3 — gap closed
    

The workload family

Kubernetes networking becomes simple when you learn it as four separate problems, each with its own mechanism.

The four networking problems

flowchart TD
    Q1["PROBLEM 1\ncontainer ↔ container\ninside one pod"]
    Q2["PROBLEM 2\npod ↔ pod\nacross nodes"]
    Q3["PROBLEM 3\npod ↔ Service\nstable address for moving pods"]
    Q4["PROBLEM 4\ninternet ↔ Service\ngetting traffic in from outside"]
    Q1 --> Q2 --> Q3 --> Q4
    

Problem 1 — Container ↔ container

Solved by the Pod: containers share a network namespace, talk over localhost:<port>.

Problem 2 — Pod ↔ pod: the flat network

Three hard rules: every pod gets a unique IP, every pod can reach every other pod without NAT, nodes can reach pods without NAT. A CNI plugin (Calico, Cilium, Flannel) supplies the plumbing.

Problem 3 — Pod ↔ Service

Your frontend doesn't track pods. It talks to a Service: a stable virtual IP + DNS name that load-balances to whichever pods currently match a label selector.

flowchart TD
    FE["frontend pod\ncurl http://payments (DNS)"] --> VIP["SERVICE 'payments'\nClusterIP 10.96.0.10 (stable, virtual)\nselector: app=payments"]
    VIP --> E1["pod 10.244.1.5"]
    VIP --> E2["pod 10.244.2.7"]
    VIP --> E3["pod 10.244.3.2"]
    

The machinery:

1. ClusterIP — a virtual IP, exists only as kernel rules.
2. EndpointSlice controller — keeps a live list of healthy pod IPs.
3. kube-proxy — programs iptables/IPVS rules for DNAT to random ready endpoints.
4. CoreDNS — resolves service names to ClusterIP.

sequenceDiagram
    participant A as frontend pod
    participant DNS as CoreDNS
    participant K as kube-proxy rules
    participant B as payments pod
    A->>DNS: resolve "payments"
    DNS-->>A: 10.96.0.10
    A->>K: connect 10.96.0.10:80
    K->>B: DNAT → 10.244.2.7:8080 (random ready endpoint)
    B-->>A: response
    

Problem 4 — Internet ↔ Service

Service types build on each other:

flowchart LR
    CI["ClusterIP (default)\nreachable only INSIDE the cluster"]
    NP["NodePort\n= ClusterIP + high port\n(30000-32767) on EVERY node"]
    LB["LoadBalancer\n= NodePort + asks the cloud\nfor a real external LB"]
    CI --> NP --> LB
    

Ingress — the smart front door

An Ingress gives you one entry point with L7 routing (host/path rules), TLS termination, and auth — instead of paying for one cloud LB per service.

flowchart TD
    NET["Internet\napi.example.com · example.com/shop"] --> IC["INGRESS CONTROLLER\n(nginx/traefik — exposed via one LoadBalancer)"]
    IC -->|"host/path rules"| S1["Service: api"]
    IC --> S2["Service: shop"]
    S1 --> P1["api pods"]
    S2 --> P2["shop pods"]
    

One request, end to end

Browser → cloud LB → Ingress controller (TLS off, route by path) → Service ClusterIP → kube-proxy DNAT → pod IP → container port.

Pods are cattle. Their filesystems die with them. Kubernetes solves this with three layers of abstraction.

Layer 0 — Volumes: outliving a container, not the pod

A Volume is a directory defined on the pod. Its lifetime = the pod's lifetime.

Layers 1–3: PV, PVC, StorageClass

flowchart LR
    subgraph CLUSTER["Cluster"]
        SC["StorageClass 'fast'\nTHE RECIPE\nhow to make a disk"]
        PV["PersistentVolume (PV)\nTHE THING\nan actual disk"]
        PVC["PersistentVolumeClaim (PVC)\nTHE REQUEST\n'I need 10 Gi, storageClass: fast'"]
    end
    POD["Pod\nmounts the PVC by name"] --> PVC
    PVC <-->|"bound 1-to-1"| PV
    SC -.->|"dynamic provisioning:\ncreates PVs on demand"| PV
    

StorageClass = the menu (defined by platform team). PV = the actual disk. PVC = the order ticket. The pod spec never mentions disks, zones, or clouds. Portability through indirection.

Dynamic provisioning

sequenceDiagram
    participant U as App team
    participant A as API Server / etcd
    participant PR as CSI provisioner
    participant CL as Storage backend
    U->>A: create PVC: 10 Gi, class "fast"
    A-->>PR: watch event: unbound PVC
    PR->>CL: create disk (10 Gi SSD)
    CL-->>PR: disk ready (id)
    PR->>A: create PV object + bind it to the PVC
    

Details that bite in production

Access modes:

Reclaim policy: Retain (keep the disk) or Delete (destroy with claim).

StatefulSets use volumeClaimTemplates so each pod gets its own stable PVC — db-0 always reattaches to data-db-0. Stable identity + stable storage.

Build one image; promote it dev → staging → prod; inject the differences at runtime. ConfigMaps and Secrets are that injection mechanism.

The two objects

Three ways to consume them

flowchart TD
    subgraph SRC["Sources (live in etcd)"]
        CM["ConfigMap\napp-config"]
        SEC["Secret\ndb-credentials"]
    end
    subgraph POD["Pod"]
        ENV["1 · Environment variables\nFROZEN at container start\n(change requires pod restart)"]
        VOL["2 · Mounted files\n/etc/config/...\nAUTO-UPDATED by kubelet\n(app must re-read files)"]
        ARG["3 · Command-line args\nbuilt from env vars"]
    end
    CM --> ENV
    CM --> VOL
    SEC --> ENV
    SEC --> VOL
    ENV --> ARG
    

Update semantics:

• env vars: injected at process start. Update ConfigMap → running pods see nothing until restarted.
• volume mounts: kubelet syncs changes into mounted files (~a minute). App must notice and reload.

YAML sketch

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  application.yml: |
    server:
      port: 8080
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.4
      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef: { name: app-config, key: LOG_LEVEL }
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: { name: db-credentials, key: password }
      volumeMounts:
        - name: config
          mountPath: /etc/app
  volumes:
    - name: config
      configMap: { name: app-config }

The scheduler's entire job: for each pending pod, choose one node and write spec.nodeName. It never starts anything — it only decides.

The scheduling pipeline

flowchart TD
    P["New pod appears\nspec.nodeName is EMPTY"] --> W["Scheduler (watching API server)\npicks it up"]
    W --> F
    subgraph F["PHASE 1 — FILTERING (hard constraints)"]
        F1["Drop nodes that can't fit:\nnot enough CPU/memory REQUESTS\n· port already taken\n· nodeSelector / affinity mismatch\n· untolerated taint\n· required volume unavailable\n· node unready"]
    end
    F --> S
    subgraph S2["PHASE 2 — SCORING (soft preferences)"]
        S1["Rank the survivors:\nspread replicas across nodes/zones\n· balance resource usage\n· prefer nodes with the image cached"]
    end
    S2 --> B["PHASE 3 — BIND\nwrite nodeName via the API server"]
    B --> K["Kubelet on chosen node takes over\n(pull image, start containers)"]
    

Requests & limits — the most important numbers you'll set

resources:
  requests:        # "reserve this for me" — used by the SCHEDULER
    cpu: "500m"    # 0.5 CPU
    memory: "256Mi"
  limits:          # "never let me exceed this" — enforced by the KERNEL
    cpu: "1"
    memory: "512Mi"

flowchart LR
    subgraph NODE["Node: 4 CPU · 8 GiB allocatable"]
        A["Pod A\nreq 1 CPU · 1 Gi\nlim 2 CPU · 2 Gi"]
        B["Pod B\nreq 500m · 2 Gi\nlim 2 · 4 Gi"]
        FREE["Scheduler's math:\nrequests used = 1.5 CPU / 3 Gi\n→ still schedulable: 2.5 CPU / 5 Gi\n(limits are NOT reserved)"]
    end
    

Enforcement at runtime

QoS classes — who dies first under pressure

flowchart TD
    subgraph ORDER["Eviction order under node pressure (first → last)"]
        BE["BestEffort\nno requests/limits at all\nevicted FIRST"]
        BU["Burstable\nrequests < limits"]
        GU["Guaranteed\nrequests == limits for every container\nevicted LAST"]
        BE --> BU --> GU
    end
    

Steering the scheduler

flowchart TD
    subgraph STEER["From simplest to most expressive"]
        NS["nodeSelector\n'only nodes labeled disk=ssd'"]
        AFF["Affinity / anti-affinity\n'prefer/require nodes in zone-a'\n'keep replicas on DIFFERENT nodes'"]
        TAINT["Taints + tolerations\nthe NODE repels pods:\n'GPU nodes only for pods that tolerate gpu=true'"]
        NS --> AFF --> TAINT
    end
    

Production rule: always set requests (so scheduling and eviction are sane); set limits thoughtfully.

Trace one command — kubectl apply -f deployment.yaml — through the entire cluster. If you can narrate these traces unaided, you understand Kubernetes.

Trace 1: kubectl apply → running pods

sequenceDiagram
    autonumber
    participant U as You (kubectl)
    participant A as API Server
    participant E as etcd
    participant DC as Deployment controller
    participant RC as ReplicaSet controller
    participant S as Scheduler
    participant K as Kubelet (node 2)
    participant R as Container runtime
    participant D as CoreDNS / kube-proxy
    U->>A: POST Deployment (YAML → JSON over HTTPS)
    A->>A: authenticate → authorize (RBAC) → admission → validate
    A->>E: persist Deployment object
    A-->>U: 201 Created — NOTE: nothing is running yet!
    E-->>DC: watch: new Deployment observed
    DC->>A: create ReplicaSet (pod template + hash)
    A->>E: persist ReplicaSet
    E-->>RC: watch: new ReplicaSet, status 0 of 3
    RC->>A: create Pod ×3 (spec.nodeName empty)
    A->>E: persist Pods
    E-->>S: watch: unscheduled pods exist
    S->>S: filter nodes → score → pick node 2
    S->>A: bind: pod.spec.nodeName = node-2
    A->>E: persist binding
    E-->>K: watch: pod assigned to MY node
    K->>R: CRI: pull image, create containers
    R-->>K: containers running
    K->>A: update pod status: Running, then Ready
    A->>E: persist status
    E-->>D: watch: new ready endpoint
    D->>D: DNS answers, kube-proxy rules updated
    

Key takeaways:

1. apply only writes desired state. Step 4 returned success before any container existed.
2. Nobody calls anybody. Each participant woke from a watch, did one thing, wrote back.
3. Every hop is persisted in etcd first. Crash any component mid-trace; its successor still sees the last committed state.
4. Ready gates traffic. The pod existed before it received Service traffic.

Trace 2: a node dies (self-healing, assembled)

sequenceDiagram
    autonumber
    participant N as Node 2 (runs 1 of 3 web pods)
    participant A as API Server / etcd
    participant NC as Node controller
    participant RC as ReplicaSet controller
    participant S as Scheduler
    participant K as Kubelet (node 1)
    N->>A: heartbeats every few seconds
    Note over N: NODE 2 DIES
    N -x A: silence
    NC->>A: mark node NotReady (after ~40s grace)
    NC->>A: evict pods (default toleration ~5 min)
    Note over RC: routine loop pass: desired 3, actual 2 → gap!
    RC->>A: create replacement pod
    S->>A: bind → node 1 (best feasible node)
    K->>K: pull image, start container
    K->>A: status Running → Ready → endpoints updated
    

No "node failure handler" exists anywhere. Recovery is just ordinary loops observing an ordinary gap. Self-healing is not a feature bolted on; it is the inevitable behavior of reconciliation loops over a shared source of truth.

The final mental model

flowchart TD
    YOU["YOU\ndeclare desired state (YAML, Git, CI)"] --> API
    subgraph K2["KUBERNETES"]
        API["API server\n(the front door)"]
        ETCD[("etcd\n(desired + observed state)")]
        LOOPS["Controllers · Scheduler · Kubelets · kube-proxy\nwatch → compare → act → repeat"]
        API <--> ETCD
        API <--> LOOPS
    end
    LOOPS --> WORLD["THE REAL WORLD\ncontainers, networks, disks, load balancers"]
    WORLD -->|"status reported back"| API
    

If you remember only five things

1. You declare desired state; the system perpetually makes it true.
2. Every object is spec vs status; the gap is the work.
3. Everything coordinates through the API server + etcd — never directly.
4. Pods are cattle: disposable, labeled, replaced — never repaired.
5. Labels + selectors wire everything: ReplicaSets → pods, Services → endpoints.

Theory is done. Now make it muscle memory: spin up a real cluster on your machine.

Object hierarchy

flowchart TD
    NS["Namespace (logical partition)"] --> D["Deployment\nrollout strategy"]
    D --> RS["ReplicaSet\n'keep N pods alive'"]
    RS --> P["Pod\nshared net + storage"]
    P --> C["Container(s)\nnamespaces + cgroups + image"]
    SVC["Service\nstable VIP + DNS"] -. "selector:\nlabels" .-> P
    ING["Ingress"] -.-> SVC
    P --> PVC["PVC → PV → disk"]
    P --> CM["ConfigMap / Secret"]
    

The lab (30 minutes)

Setup

brew install kind kubectl
kind create cluster --name learn
kubectl cluster-info
kubectl get nodes
kubectl get pods -n kube-system

Act 1 — desired state & self-healing

kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl get pods -w                # watch Pending → Running
kubectl get rs,deploy
kubectl delete pod <one-pod-name>  # kill one...
kubectl get pods                   # ...a replacement is already Running!

Act 2 — rollouts

kubectl set image deployment/web nginx=nginx:1.28
kubectl rollout status deployment/web
kubectl get rs                     # TWO ReplicaSets: old (0), new (3)
kubectl rollout undo deployment/web

Act 3 — services & DNS

kubectl expose deployment web --port=80
kubectl get svc web
kubectl run curl --rm -it --image=curlimages/curl -- sh
# inside the pod:
curl web                           # DNS name resolves!
curl web.default.svc.cluster.local # full form
exit

Act 4 — resources & scheduling

kubectl set resources deployment/web --requests=cpu=100m,memory=64Mi --limits=memory=128Mi
kubectl describe node | grep -A5 "Allocated resources"
kubectl scale deployment web --replicas=5
kubectl get pods -o wide

Act 5 — break it on purpose

kubectl set image deployment/web nginx=nginx:doesnotexist
kubectl get pods          # ImagePullBackOff
kubectl rollout status deployment/web   # stuck — old pods STILL RUNNING (safe!)
kubectl rollout undo deployment/web     # gap closed

Cleanup

kind delete cluster --name learn

kubectl survival card

Debugging decision tree

flowchart TD
    X["Something's wrong"] --> Q1{"Pod exists?"}
    Q1 -->|"no"| A1["Check the controller:\ndescribe deploy/rs → events"]
    Q1 -->|"Pending"| A2["describe pod → scheduling failure?\nrequests too big? taints? PVC unbound?"]
    Q1 -->|"ImagePullBackOff"| A3["image name/tag wrong?\nprivate registry credentials?"]
    Q1 -->|"CrashLoopBackOff"| A4["logs --previous → app crashes on start?\nconfig missing?"]
    Q1 -->|"Running but no traffic"| A5["endpoints empty?\nselector labels match pod labels?\nreadiness probe passing?"]
    

Glossary

Where next

• Practice: rebuild the lab from memory, then add a ConfigMap, a PVC, an Ingress.
• Helm / Kustomize — packaging and templating YAML.
• GitOps (Argo CD / Flux) — desired state living in Git, reconciled continuously.
• Observability — Prometheus + Grafana; understand your loops' telemetry.
• CKA curriculum — if you want a certification target.