How to Effectively Communicate Commonly misinterpreted Statistical terms in experimentation AMAZON IPC LAB

“To invent you have to experiment, and if you know in advance that it's going to work, it's not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.” – Jeff Bezos Culture of experimentation at Amazon

Amazon’s Supply CHAIN Treatment effect:

Experimentation IN SUPPLY CHAIN Randomized Controlled Trials in production Customers Teams in Supply Chain Optimization Technologies (SCOT) and Retail organizations within Amazon Software Developers, Product Managers, Research Scientists, Senior Leaders A/B Testing ?

Measuring treatment effects Treatment Effect: Impact of a new idea, or changes in supply chain. Method: OLS regression What do we provide to the customers? Treatment effect estimates 95% confidence intervals

Challenges Lack of intuitive explanation of statistical theories for customers Customers’ tendency to interpret results based on point estimates, without considering uncertainty measures “The tests themselves give no final verdict, but as tools help the worker who is using them to form his final decision.” - Neyman and Pearson

Commonly misinterpreted terms Correlation ⍯ causation Spurious Correlations Concept of unobserved factors or confounders Absence of evidence ⍯ evidence of absence Goal of the experiment ⍯ achieve significance Interim Results Peeking Influence of early adopters on results

Commonly misinterpreted terms Interpretation of Uncertainty Power and Minimum Detectable Effect p-value 95% Confidence Intervals Difference “on-average” Estimate = Expected value of unknown population parameter Only one of the potential outcome observed for each individual Simpson’s Paradox Opposite directions for observed effect for all subjects as a single group vs. separately for each group

Ongoing WORK for effective communication Technical FAQ page Visualization and dashboards Bayesian Probabilistic Statements Introduce Type S (sign) and Type M (magnitude) errors in results Training and awareness Certification and bar-raisers

Questions? Thank you!