1. Testing in Production Key to Data Driven Quality
#GHC14
Testing in Production Key to Data Driven Quality
Jyoti Jacob
Senior Software Engineer- Microsoft
10/9/2014
2014

2. Agenda Why and What is Testing in Production? Types
Analytics Vs Synthetics
Flighting and Experimentation
TiP in other companies
My Key Takeaways

3. Traditional Software Development
Scrub dates
Longer Deployment cycles
Credits: Patrick Patterson
Director of Test, APPs, Microsoft

4. Challenges of Service Meet Availability SLA Good customer experience
Faster detection
Faster Recovery
Failure not option but inevitable
Unpredictable user interactions
Environment and partner dependencies
Agility = faster deployment
Learn from service
Easier to deploy
Easier to get to the customer
Telemetry Data
Production difficult to mimic

5. Development in Service
Design
Scenarios
Metrics
Experimentation
Develop
Implementation
Validation
Deploy
TiP
Flighting
Evaluate
Analyze
Adjust
Data Driven Decision
Continuous Development
Faster Deployment
Availability with agility
Each feature is released separately.
Credits: Patrick Patterson
Director of Test, APPs, Microsoft

6. Testing in Production(TiP)
Testing in production (TiP) is a set of software methodologies that derive quality assessments not from test results run in a lab but from where your services actually run – in production. 
Seth Eliot, Principal Knowledge Engineer, Microsoft

7. Types of Testing

8. Robot initiated actions for availability
Synthetics
Robot initiated actions for availability
Goal is Availability
Simulate customer scenarios
Should trigger alerts on failure.
E.g. active monitoring, customer simulation, fault injection etc.
Useful for maintaining SLA
Alert
Probe
Animation of alert and tag together
Triggered few mins
Production synthetic transaction HomePage availability drops to 4% in content farm <farm info>

9. Data driven validation
Analytics
e.g. API failed for specific locale on unknown browser
Data driven validation
Data analysis and alerts based on conditions.
Real users mostly == varying actions.
Measure true customer experience
Analyzing logs and data
Alerts- not always urgent. Could be based on threshold or occurring over a period or condition.
PII scrubbed logs
-Request start
-Success
-Failure
-Perf latency
Device image
Define machine learning
Alert
Detect Issues
Issues, Perf and Usage Analysis

10. Analytics Dashboard

11. API Analysis Example

12. Feature Flighting
Deployment

13. Deployment (a. k. a feature toggle, a. k
Deployment (a.k.a feature toggle, a.k.a deployment does not equate release)
If #featureEnabled {
Do Something; }
You can check-in features to Production but the code path will never be hit if the feature is not enabled.
Rethink image
Test
Production
Big Red Switch
Pre-Production

14. Feature Feedback
Even during Design

15. A/B Testing (a.k.a Online Controlled Experimentation)
Most of the users get the original experience (control)
Some users are offered the new experiences or features (experiment)
For success:
least # of variables should change.
Remove random noise or assumption.
Detrimental variable are masked
Data shows that when ice cream consumption increases , drowing increases

16. TiP in Other Companies Data driven quality
“Netflix is a log generating company that also happens to streams movies”- Adrian Cockroft
A/B and multi-variate testing for Experimentations.
Have adopted some form of dark or ramped deployment.
Shadowing e.g. Google

17. My Key Takeaways What not to do Key Learnings Expose PII information
Too many synthetic transactions
Expose test data to customers.
Key Learnings
“It is a capital mistake to theorize before one has data” –Sherlock Holmes
Engineers need access to data and debug boxes easily but with security considerations.
Synthetics + Analytics + measurements against key performance indicators (KPI) = Quality Assessments

18. Questions
@ end of session.

19. Got Feedback? Rate and Review the session using the GHC Mobile App
To download visit
This is the last slide and must be included in the slide deck