1046: Weighing a Feather: How to Design A/B Tests That Work, with Billur Engin
In Episode #1046, Billur Engin (who leads data science and analytics at Prolific) joins Jon Krohn to explain how to design experiments that actually teach you something. Too many A/B tests, she says, are like bathroom scales trying to weigh a feather the effect is real, but the system was never built to detect it. In her first-ever podcast appearance, Billur walks through why complexity has to earn its place in your models, the biggest myth about A/B testing, the AA test she runs to catch broken measurement, three ways to get at causality when you can’t run a controlled experiment, and why causal inference matters more than ever now that prediction has become cheap. Interested in sponsoring a SuperDataScience Podcast episode? Email [email protected] for sponsorship information. In this episode you will learn: (01:18) What Prolific does and why representative data matters (05:19) Why complexity has to earn its place (07:29) The biggest myth about A/B testing (11:34) Why causal inference matters more than ever (14:39) Getting at causality when you can’t run an experiment
