DevOps training with projects: break, fix, write the runbook

The best way to practise DevOps without a job in it is a loop you run on purpose: break one thing in a small lab, find it from logs and metrics alone, fix it, then write the runbook you wish you'd had. Good devops training with projects ends each lab with that page, not with a green tick. Google trains new site reliability engineers this way, with break-fix exercises and role-played outages ¹. Even so, most of its SRE teams don't expect a new hire to be ready for on-call before three to nine months ². A home lab won't shorten that. It gives you the practice.

Why breaking things teaches more than building them

Most self-study projects stop the moment the app comes up. That's the easy part of the job. The hard part is the afternoon when it doesn't, and someone is waiting for you.

Google's SRE book names its training method plainly: "Break Real Things, Fix Real Things" ¹. Trainees repair a diverted instance or a staging stack under synthetic load. In one variant they begin with a known-good setup, predict how it will react, and only then degrade it at chosen bottlenecks ¹. The prediction is the useful bit. It shows you where your mental model of the system is wrong.

The same chapter describes "disaster role playing", which teams call the Wheel of Misfortune. A facilitator presents a scenario, and the on-call engineer talks through the response while the group watches ¹. You can run this with one study partner and a video call.

DevOps training with projects: the break-fix-runbook loop

Run one fault per session. Six steps:

1. Write down what healthy looks like. Which pods run, which port answers, what the dashboard normally shows. Then predict what the fault will do.
2. Break one thing. Better still, have a partner break it without telling you what they did.
3. Diagnose from the surfaces only. Logs, kubectl describe, metrics. Google's advice is to teach people the "diagnostic and debugging surfaces" of a system ¹.
4. Fix it, then prove it's fixed. Restarting until the error goes away isn't a fix.
5. Write the runbook. Symptom, the checks in order, the commands, the fix, how to verify, and what alert would have caught it sooner. Google's workbook says playbooks "reduce stress, the mean time to repair (MTTR), and the risk of human error" ².
6. Come back to it later. Run the same fault two or three weeks on, using only your runbook.

That last step matters. A 2025 meta-analysis found a small to medium benefit from spaced over massed practice (g = 0.28 across 27 studies) ⁹. It covers maths learning, and the authors say the effect may be smaller there than in other fields, so read it as direction rather than a promise. The same review found no reliable testing effect in maths ⁹. I still use re-runs because they show whether the runbook works without me in my head.

One more rule from the workbook: if a runbook has become a fixed list of commands, automate it ². Turning your third runbook into a Bash script or an Ansible playbook is a project in itself.

A home lab in four stages

You don't need a rack. Begin on the laptop you have and add hardware only when a stage asks for it.

Stage	What you run	Faults worth practising
1. One machine	A Linux VM, Docker and Compose: an app, a database, Nginx	Full disk, a wrong environment variable, a port clash, a stopped database
2. A cluster on the laptop	kind, which runs Kubernetes nodes as Docker containers and supports multi-node clusters 4	A misspelled image name, resource requests too large to schedule
3. Spare hardware	K3s on an old laptop or a Raspberry Pi, with Prometheus and Grafana, deployed through GitHub Actions and ArgoCD	A broken deploy you roll back, an alert that never fires
4. A little cloud	Terraform on AWS, built and destroyed in the same session	An IAM permission missing, a security group that blocks traffic

Stage 2 fits the Kubernetes docs closely. A pod stuck in Pending "can not be scheduled onto a node", often because CPU or memory is exhausted. A pod stuck in Waiting has a node but can't run, and the most common cause is the image: a wrong name, or an image never pushed to the registry ³. Cause both on purpose, then write the runbook for each. If you're not yet sure your work needs Kubernetes at all, read Docker or Kubernetes first? How to decide from your own workload before you build stage 2.

For stage 3, K3s asks for at least 2 CPU cores and 2 GB of RAM on the server node, and 1 core and 512 MB on an agent, before your workloads ⁵. On a Raspberry Pi the docs recommend an external SSD, because SD cards can't keep up with etcd's writes ⁵.

For stage 4, know what the free plan is. A new AWS account on the free plan runs for up to six months, or until the credits are used, whichever comes first, and some services aren't available on it ⁶. On the paid plan, anything beyond your credits is billed at standard rates ⁶. Run terraform destroy at the end of every session and write that step into your runbook.

Using AI in the lab, and checking it

An AI assistant is useful at step 3, as long as you treat it as a colleague with ideas, not as the person holding the pager. DORA's 2025 report found 90% of respondents use AI at work, and 30% report little or no trust in the code it generates ⁷. In Stack Overflow's 2025 survey, 66% of developers named "AI solutions that are almost right, but not quite" as their top frustration ⁸.

In a lab, "almost right" does no lasting harm. Ask the assistant for hypotheses, not commands. Run anything it suggests against the lab, never against a machine you care about. When it's wrong, write that in the runbook too: "suggested deleting the PVC; that loses the data".

Keep a lab log

Keep everything in one Git repository: the Compose files, the manifests, the Terraform, a runbooks/ folder and a short postmortem for each fault. Google suggests reading other people's postmortems in "tales of fail" sessions to build a map of how systems fail ¹.

The log does two jobs. It shows your progress, and it is something concrete you can talk through in an interview. It doesn't stand in for production experience, and no honest reviewer will treat it as if it did.

The DevOps Engineer course at TTFA covers this ground in its Kubernetes troubleshooting, monitoring and SRE topics, with a module on checking AI output before it reaches production.

What we don't know

There's no good study of how home-lab practice turns into on-the-job performance in DevOps. The spacing evidence comes from maths classrooms ⁹, and Google's training methods come from 2017 and 2018 books written for people who already have the job ¹². Treat the loop as a sound method rather than a measured one.

What to do this week: build stage 1, break it three different ways, and write three runbooks. Next in this series: platform engineering and AI in the pipeline, what changed in DevOps this year, and which parts you can safely ignore for now.

Frequently asked questions

Do I need to pay for cloud to practise DevOps?

Not at first. Stages 1 to 3 run on a laptop or spare hardware with free tools such as kind and K3s ⁴⁵. When you move to AWS, the free plan lasts up to six months or until the credits run out ⁶, so destroy what you build after each session.

What is a runbook, and how long should it be?

It's a short page telling the person on call what to check and do for one symptom. Google's workbook describes playbooks as high-level instructions for responding to alerts ². If yours runs past a page, split it by symptom.

How often should I run the practice loop?

My suggestion is one fault per session and two sessions a week, with a re-run of an old fault every couple of weeks. The spacing evidence supports coming back to material rather than cramming it ⁹.

Can I use ChatGPT or Copilot to fix my lab?

Yes, for ideas. Check every command against the lab before you trust it: 66% of developers in Stack Overflow's 2025 survey named "almost right, but not quite" answers as their top frustration with AI tools ⁸.