Waterfall: the mock-exam simulator I built while studying for OTCA
In my last post about passing the OpenTelemetry Certified Associate exam, I mentioned a side project I’d built to train instead of buying someone else’s practice tests, and promised a dedicated write-up on it. Here it is.
The project is called Waterfall, it’s live at waterfall.smartbamboo.net, and it’s a mock-exam simulator for IT certifications, currently covering OTCA, with CKAD and Terraform Associate next on the list.
The name
If you’ve spent any time in an observability tool, you know the waterfall view: a trace broken down into spans, each one a horizontal bar showing where time went and in what order. It’s the diagram you stare at when you’re debugging latency. OTCA, the first exam I built this simulator for, is entirely about that mental model: traces, spans, context propagation. The name stuck.
How I used to prepare for exams
Before reaching for an AI agent for anything like this, my certification prep followed the same loop every time, and I used it for the CKAD, CKA, and AWS SAA:
- Go over as many blogs as I can find, meaning other candidates’ feedback posts, community write-ups, and forum threads, to figure out what actually shows up on the exam and what people consistently get wrong.
- Buy practice material, meaning Udemy courses, exam simulators, whatever paid platform had decent reviews, for structured training and mock exams close to the real format.
- Sit the real exam once I’m consistently scoring well above the passing bar in practice.
That process works, but step 2 is where it gets expensive and generic: you’re training on someone else’s question bank, written for the broadest possible audience, not necessarily matching this exam’s current domain weights or your own gaps.
Same idea, built instead of bought
Like a lot of people right now, I’ve been spending a good chunk of my time building things with AI coding agents, Claude Code in particular. While prepping for OTCA, I realized I could fold the two together: keep step 1 exactly as before (scan blogs and community feedback for what the exam actually tests), but replace step 2 with something I build myself instead of paying for.
So that’s what Waterfall is: the tedious part, searching for sample questions, digesting candidate feedback, mapping it all against the official curriculum and domain weights, is handled with the agent’s help. What comes out the other end is a question bank plus a small app to train against it, shaped exactly like the real exam instead of a generic one.
One constraint mattered to me from the start: every question is authored from the official curriculum, nothing is copied from commercial dump sites. The point was never to leak exam content; it was to build well-designed original questions that test the same domains, in the same proportions, under the same conditions.
Why free material out there falls short
Free practice content for cloud-native certifications exists, but it’s scattered across gists, blog comments, and low-effort dump sites, and most of it doesn’t reproduce the actual exam shape: wrong question count, no timer, no domain weighting, and rarely an explanation of why an answer is correct. Waterfall’s whole premise is to fix the shape and treat explanations as the primary deliverable, not an afterthought.
What it looks like today
Opening a certification in Waterfall gives you the exam’s actual metadata up front: question count, time limit, passing score, curriculum version, and how many questions are currently in the bank, before you commit to a session. For OTCA right now that’s 60 questions, 90 minutes, 75% to pass, drawn from a bank of 150.
From there you pick a session length:
- Quick mock: 20 questions, untimed
- Half paper: 30 questions, untimed
- Full exam: 60 questions, 90 minutes, matching the real conditions exactly
- Whole bank: all 150 questions, untimed, for deep review
Session length is just one axis, and the filters stack: on top of it you can scope a session to specific domains (Fundamentals of Observability, the API and SDK, the Collector, Maintaining and Debugging Pipelines for OTCA) and a specific difficulty level and a session length, all at once, a half exam on nothing but “API and SDK” at difficulty 3, for instance, if that’s the domain that’s giving you trouble. There’s also an option to match the official curriculum weighting rather than draw uniformly from the filtered pool, which is exactly the “revise by domain percentage, not by comfort” lesson from my OTCA post, baked directly into the tool.
Once a session starts, the exam screen itself mirrors what a proctored exam actually feels like: each question carries its domain and difficulty tag, a running timer counts down in the header, and a Navigator panel on the side shows every question in the sitting as a grid: answered, flagged, or unseen, so you can jump around instead of answering strictly in order. Flag anything you’re unsure of for a second pass, then submit the exam when you’re done. It’s a small thing, but rehearsing that navigation muscle memory, jumping between questions, tracking what you’ve flagged, watching the clock, is exactly the kind of pressure a static PDF of practice questions can’t reproduce.
Features, phase 1–2
- Multiple exams from a single install, each with its own domain model and timing
- Exam mode at the real paper length, plus shorter mock and half-paper sittings
- Domain-weighted question selection matching the official curriculum weights
- Flashcard mode over the full bank, self-graded
- Review mode: wrong answers only, with explanations and links to primary sources
- Every session reshuffled, questions and options, and reproducible from a seed
- Attempt history and per-question stats in local storage
- JSON import/export of your progress
Planned
- Weak-spot drilling driven by spaced repetition
- Progress analytics across attempts
- Firebase Auth + Firestore sync across devices
- Authoring CLI for faster bank ingestion
Why the engine matters more than any single exam
The part I’m most interested in going forward isn’t the OTCA bank; it’s that nothing about the app is OTCA-specific. Exams, domains, questions, and weighting are all data, not code. Adding CKAD or Terraform Associate is a matter of producing a new question bank against their curricula, not touching the simulator itself. Given how many certifications I seem to keep signing up for, a reusable engine is probably the most valuable output of this whole project.
Try it
Waterfall is live at waterfall.smartbamboo.net. If you’re prepping for OTCA, go run a full timed exam under real conditions before you book the actual one; that’s exactly the gap it’s meant to close.
I’ll follow up with a deeper technical post on the build itself: how the question bank gets generated and reviewed, and the actual stack behind the app, once CKAD support lands.
References:
