Learning goal
After this lesson you can define in advance how you will recognize an initiative’s impact – with baseline, definition, comparison period and cohort. And you know when you may say “it worked” and when only “it changed.”
Four prerequisites
- Baseline
- The value of the metric before the measure, measured, not remembered. Without a baseline there is no comparison, only an impression.
- Definition
- Numerator, denominator, period, source. If the definition changes between before and after, you’re measuring the definition, not the initiative.
- Comparison period
- Same length, comparable season, completed. One quarter against two weeks is not a comparison.
- Cohort
- Cases that started within a period and are tracked until a cutoff date. Only this way do you compare paths, not snapshots.
Result instead of activity
“We followed up on 40 quotes” is activity. “Of 40 quotes followed up on, 12 were decided within 14 days” is a result. Activity is easy to measure and feels good. Results are what the customer does. Measure both, but don’t confuse them – and let only the result decide whether the initiative succeeded.
An example
A software company introduces a rule in April: Every inquiry gets a personal reply within four working hours. In June the team reports: “Closes are up 30%.” Sounds good. The check: Baseline? In the first quarter, 9 of 120 inquiries were won, so 7.5%. Definition? Closes relative to inquiries of the cohort, measured 60 days after receipt. Comparison period? The April–May cohort, measured at the end of July: 13 of 130, so 10%. That's an improvement from 7.5% to 10%, about 33% relative but 2.5 percentage points absolute.
Did the rule work? The number changed. But in the same period the price list was also simplified and a new salesperson started. Without a design that separates these influences – such as a comparison group the rule did not apply to, or a staggered start by team – “the response time increased closings” is a guess. A plausible guess, but not a proven one. The honest sentence reads: “The cohort’s closing rate rose from 7.5% to 10%; the response rule is one of three changes in the period.”
When you may claim causality
- When there was a comparison group that did not receive the initiative and was otherwise comparable.
- When the initiative was rolled out in a staggered way and the effect appeared each time with the rollout.
- When you withdrew the initiative and the effect disappeared.
Without such a design, you describe the change and name the competing explanations. That's not an admission of weakness but the prerequisite for the next decision to rest on something solid.
Exercise
- 01Choose an initiative that is running or planned.
- 02Write down the baseline, definition (numerator, denominator, period, source), comparison period and cohort – before you look at the result.
- 03Note which other changes are happening in the same period.
- 04Write down in advance the sentence you'll say on success, without the word “because,” unless you have a design for it.
Reflection aid
- Would you have had the baseline without this lesson? If not, every success report so far was an impression.
- Did you state absolute and relative change separately? 33% relative and 2.5 percentage points absolute are the same number – and look completely different.
- Which competing explanation is the strongest – and can you test it with the data you have?
This ends the self-study path: a hypothesis, a weekly list, clean deals, a calculated automation, and a measurement design. How to build your own benchmarks from this is described in “Build your own benchmarks.”
