Statistical Methods in Online A/B Testing

“Statistical Methods in Online A/B Testing” is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, that tackles the difficult matter of statistical inference in a way accessible to readers with little to no prior experience with it. Each concept is built from the ground up, explained thoroughly, and illustrated with practical examples from website testing.

The presentation is straight to the point and practically oriented so you can apply the takeaways in your daily work. It is a must-read for anyone looking for a deep understanding of how to make data-driven business decisions through experimentation: conversion rate optimizers, product managers, growth experts, data analysts, marketing managers, experts in user experience and design.

The new research presented and the fresh perspective on how to apply statistics and experimentation to achieve business goals make for an interesting read even for experienced statisticians.The book deals with scientific methods, but their introductions and explanations are grounded in the business goals they help achieve, such as innovating under controlled risk, and estimating the effect of proposed business actions before committing to them. While the book doesn’t shy away from math and formulas, it is to the extent to which these are essential for understanding and applying the underlying concepts. The presentation is friendly to readers with little to no prior knowledge in statistics. Artificial and impractical examples like dice rolling and betting are absent, instead statistical concepts are illustrated through scenarios which might well be mistaken with the last couple of A/B tests you managed.

This book also doesn’t shy away from the fact that much of the current statistical theory and practice in online A/B testing is misguided, misinterpreted, or misapplied. It also addresses the issue of blind copying of scientific applications without due consideration of the unique features of online business, which is widespread. The book will help you avoid these malpractices by explicitly pointing out frequent mistakes, while also helping you align your usage of statistics and experimentation with any business goals you might want to pursue.