1. How to Effectively Communicate Commonly misinterpreted Statistical terms in experimentation
AMAZON IPC LAB

2. “To invent you have to experiment, and if you know in advance that it's going to work, it's not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.” – Jeff Bezos
Culture of experimentation at Amazon

3. Amazon’s Supply CHAIN
Treatment effect:

4. Experimentation IN SUPPLY CHAIN
Randomized Controlled Trials in production
Customers
Teams in Supply Chain Optimization Technologies (SCOT) and Retail organizations within Amazon
Software Developers, Product Managers, Research Scientists, Senior Leaders
A/B Testing ?

5. Measuring treatment effects
Treatment Effect: Impact of a new idea, or changes in supply chain.
Method: OLS regression
What do we provide to the customers?
Treatment effect estimates
95% confidence intervals

6. Challenges
Lack of intuitive explanation of statistical theories for customers
Customers’ tendency to interpret results based on point estimates, without considering uncertainty measures
“The tests themselves give no final verdict, but as tools help the worker who is using them to form his final decision.” - Neyman and Pearson

7. Commonly misinterpreted terms
Correlation ⍯ causation
Spurious Correlations
Concept of unobserved factors or confounders
Absence of evidence ⍯ evidence of absence
Goal of the experiment ⍯ achieve significance
Interim Results
Peeking
Influence of early adopters on results

8. Commonly misinterpreted terms
Interpretation of Uncertainty
Power and Minimum Detectable Effect
p-value
95% Confidence Intervals
Difference “on-average”
Estimate = Expected value of unknown population parameter
Only one of the potential outcome observed for each individual
Simpson’s Paradox
Opposite directions for observed effect for all subjects as a single group vs. separately for each group

9. Ongoing WORK for effective communication
Technical FAQ page
Visualization and dashboards
Bayesian Probabilistic Statements
Introduce Type S (sign) and Type M (magnitude) errors in results
Training and awareness
Certification and bar-raisers

10. Questions? Thank you!