How I read a number
ยท Method
Three checks, in order, before anything gets charted. They are dull and they catch most of it.
1. What exactly is counted
Not what the label says. What the definition says.
"Active users" is four different numbers depending on whether it means signed in, took an action, took a meaningful action, or was merely loaded by a background tab. "Pass rate" depends on whether a skipped case counts as a pass, a fail, or nothing. None of those choices is wrong; all of them are different, and a comparison across two of them is not a comparison.
This is also where the definition has to be fixed before the number arrives. A definition adjusted after the result is a definition chosen to produce the result.
2. What is excluded, and whether the exclusion is load-bearing
Every number excludes something. The question is whether the excluded part is the interesting part.
A conversion rate that excludes traffic from one source is a fine number until that source is the one that changed. A latency p95 that excludes timeouts is measuring how fast the requests that finished were, which is a different question from how fast the system is, and the gap between the two is usually the whole story.
3. What else changed at the same time
The confound. Almost always there is one, and almost always it is knowable with ten minutes of looking.
The rule I try to hold to is that the confound goes in the write-up rather than in a footnote, and it goes in even when it does not overturn the conclusion. A memo that says "B beat A by 12%, and we also changed the email that week, and I still think it was B" is more useful than one that says "B beat A by 12%". The first one can be argued with. The second one can only be believed or not.
Why the tools look the way they do
The three apps on this desk are all built around the same idea: the method has to be as visible as the result.
Sift profiles every column of a dropped CSV before you trust it, because check one is easier to do when something has already counted the nulls and the distinct values for you.
The AB testing tool leads with the verdict and then shows the two-proportion z-test with the numbers substituted into it, because a p-value with no visible method is a number people either defer to or dismiss, and neither is reading it.
The Bayes tool draws a thousand people as dots, because a 90% accurate test on a 1% base rate is a sentence people nod along to and a picture people actually understand.
The last two are built but not live, so they are not linked yet.
The standing rule
A chart with no decision attached does not go on this desk. If the number did not change what anyone did, it is not a finding, it is a fact about a database.