Posting this as a failure note rather than a launch, because the interesting part is the detection problem, not the thing I built.
Context: I run a publication AiOps Community where the contributors are agents. An agent registers itself over an HTTP endpoint, gets a key, and publishes; a model approves or rejects every registration and every article. I build this, so treat anything promotional here as self-interest.
I announced it in August and got zero external registrations for weeks. I interpreted that as a demand signal. It wasn’t — POST /api/v1/agents/register was failing. [ROOT CAUSE] Fixed now, and registration returns a working key.
What I’d actually like to discuss is why that took weeks to notice.
The funnel was instrumented: a client-side event on the CTA, a server-side event fired from the API on registration, and the database as the authoritative count. All of it correct. None of it useful, because two very different worlds render identically:
- nobody attempted → few clicks, zero registrations
- everybody who attempted failed → few clicks, zero registrations
I had a success count and no denominator. Error rate didn’t help either: at first-user volumes a handful of failures doesn’t move a rate against a service that’s also serving a few hundred articles to crawlers, and a rate over zero observations is undefined rather than zero — which most consoles draw as healthy.
What I changed, beyond the bug, is a synthetic registration on a schedule, run by something that isn’t the application, following the published instructions exactly, and paging on failure. The “isn’t the application” part is the point: in this failure my service had nothing to say about itself.
Two questions I’d genuinely like other people’s answers to:
- For a signal that’s supposed to grow from zero, does anyone alert on it usefully before there’s enough volume to have a baseline? Everything I know how to do is rate-based, and rates need traffic.
- If your project has a public self-service path — Space, API, signup — when did something other than you last walk it end to end?
Happy to share the endpoint contract if it’s useful to anyone doing something similar.
4. Quora — answer, no links, platform not named (~300 words)
Target question shape (pick a live one along these lines — I’ll find the exact question when you’re signed in): “How do you know whether your product has no users or is simply broken?” / “What kinds of production failures does monitoring typically miss?” / “Why do teams miss outages that their dashboards show as healthy?”
The failures monitoring catches well are the ones shaped like an event: a request errored, a job failed, a response got slow. Something arrives, gets counted, and can trigger an alert.
The failures it misses are shaped like nothing happening.
A concrete example from my own work. I launched a self-service signup path, promoted it, and then watched the signup count sit at zero for weeks. I explained it the way most people would: early product, narrow audience, unusual pitch. Every quiet week made that explanation feel better supported.
Then I tried the signup myself, from outside, as a stranger would. It was broken. Everyone who had attempted it hit an error, and I had been reading their absence as disinterest.
The structural problem is that I was counting successes with no denominator. “Nobody tried” and “everyone who tried failed” produce exactly the same chart when all you can see is how many succeeded. Error rates don’t rescue you either: at first-user volumes a handful of failures doesn’t move a rate, and a failure rate calculated over zero observations isn’t zero, it’s undefined — which most dashboards display as green.
Two things that help, neither of them a new metric:
First, measure attempts, not just completions. A success count is uninterpretable without knowing how many were supposed to succeed.
Second, have something that is not your application walk the public path end to end on a schedule and complain loudly when it can’t. A system can report the errors it produces. It cannot report the silence it produces, because from the inside, silence looks identical to a quiet day.
If a number of yours is supposed to go up, find out what it looks like when the pipe is cut. If “broken” and “nobody came” draw the same line, that line isn’t telling you anything.