Measuring Change
In my post Trust Your Instruments, I raised the issue of whether you should believe your intuition about change or whether you should have solid metrics – and a set of them so you can cross-check. What I didn’t explain was how to be careful about how you measure the impact on change. For that, we need to understand a bit about statistics – but don’t be concerned. Some parts of statistics are easy and powerful.
Base Rate
Base rate is the probability of something happening without any intervention. For instance, some amount of water will evaporate with a given set of conditions. Factors include the surface area, amount of time, temperature, pressure, and the amount of water vapor already in the air, called relative humidity. I can establish a rate of evaporation if I know all these variables. When we’re measuring change, we want to know the difference between the base rate and the rate we observe in the presence of our changes.
Someone recently told me, “100% of the people treated for COVID-19 with ivermectin at the clinic recovered.” On the surface, this sounds like phenomenal results. 100% cure rate. However, base rate tells us that the probability of a person not recovering from COVID-19 with no intervention is very low. The numbers vary, but let’s just say that you’ve got a 2% chance of being hospitalized for a COVID infection – and an even lower rate of mortality.
Given a perfect sample of 100 people with COVID-19, it’s likely that two will be hospitalized – but due to randomness, to be sure that you’d see at least one patient who gets hospitalized is much larger. The probability of seeing a hospitalization in 10 patients is ~18%. For 50 patients, the confidence moves to ~63% that we’d see one hospitalization. To get to a 90% confidence that you’d see even 1 hospitalization takes 114 people (given our 2% hospitalization rate). So, if you’re not at 114 people in the clinic, you can’t be reasonably confident you should have seen one hospitalization without treatment. The math gets more complicated from here, but to show a reduction of 1% hospitalization rate with 90% confidence, you’d need 670 people.
For those interested in the probability math, the formula is:
ObsAtLeast1Event = 1 – (1 – prob)quantity
In our case, probability is 0.02.
The above statement has no way of proving that the treatment is effective, because a single clinic has nearly no hope of treating 670 people with ivermectin and showing less than 3 of them getting admitted to the hospital.
Uneven Distribution
Admittedly, this is a simple version of the problem designed to expose how people don’t think about base rates correctly. This works because we’re assuming a perfect sample, and the problem with this is that a sample of people at any given clinic will be non-perfect to the problem being studied. The greater the deviance of the sample from the normal, the greater the number of participants to remove external influences from the results.
In our work with change, we need to be very clear not about the number of events that we are or are not seeing but rather about how we’d know whether those results are statistically significant.
Sometimes, when we plan our metrics, we fail to realize how much – or how little – change will be required to achieve the metrics we’re looking for.
