
Experimentation & Evidence Decisions · Module 2 of 4
The Underpowered Test
You are Maya, Senior PM at NorthPeak. Last quarter you held the onboarding redesign because a retention guardrail broke under the headline. Engineering fixed it, the clean re-test closed on day 42, and it came back flat: activation 40.00% against 42.00%, a +5.00% relative lift at confidence 0.62, under the 0.80 bar. The room has already written the post-mortem. Nobody has asked what this test could have detected. On the bench: a measuring scale with the Floor across it as a band of frosted glass, the brass bar still in its cradle, a crank, a tray of five lenses and four doors. Drag the bar on, turn the crank, seat a lens - none of it commits anything, and all of it is optional. What you find is that the smallest effect this test could ever resolve is 14.00%, against a bar of 5.00%: the number everyone is quoting sits under the glass. Wind the crank to the end of the quarter and it stops at 6.60%, still above the bar - the clock does not arrive. Seat the 24-hour lens and the fog lifts to 4.05% on data already collected, with no extra weeks at all. Then roll the change through one of four doors: call it, extend, re-instrument, or ship anyway. Discover why re-instrumenting beats the whole quarter (+$61.90k against -$90.00k), why the patient reflex costs nearly two and a half times what the impatient one does, and why the identical flat number is worthless against a 5% bar and decisive against a 20% one.
0 of 22 items completed0%~90 min remaining
Material · 22 items