1. Evaluating models for a needle in a haystack: Applications in predictive maintenance
Danielle Dean (Microsoft), Data Science Lead
@danielleodean
Shaheen Gauher (Microsoft), Data Scientist
@Shaheen_Gauher

2. Outline Predictive Maintenance Use Cases Data Science Process
Modeling & Evaluation
Random Guess, Weighted Guess (by distribution, by threshold), Majority Class
Cost factor

3. Predictive Maintenance Concepts
Predictive Maintenance in IoT
Traditional Predicative Maintenance
Goal
Improve production and/or maintenance efficiency
Ensure the reliability of machine operation
Data
Data stream (time varying features), Multiple data sources
Very limited time varying features
Scope
Component level, System level
Parts level
Approach
Data driven
Model driven
Tasks
Failure prediction, fault/failure detection & diagnosis, maintenance actions recommendation, etc. Essentially any task that improves production/maintenance efficiency
Failure prediction (prognosis), fault/failure detection & diagnosis (diagnosis)

4. Predictive Maintenance Use Cases
5/6/ :10 AM
Predictive Maintenance Use Cases
Predict the possibility of failure of an asset in the near future so that the assets can be monitored proactively to take action before the failures occur.
Aerospace
Utilities
Manufacturing
Transportation & Logistics
What is the likelihood of delay due to mechanical issues?
When is my solar panel or wind turbine going to fail next?
Will the component pass the next stage of testing on factory floor or do I need to rework?
Should I replace the brakes in my car now, or can I wait for another month?
When is this aircraft component likely to fail next?
Which circuit breakers in my system are likely to fail in the next month?
What is the root cause of the test failure?
What maintenance task should I perform on my elevator?
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Is the ATM going to dispense the next 5 notes without failing?

5. Predictive Maintenance – Qantas Airways
~24,000 sensors
50%
/year
per A380
Technical Delays
Technical Delays
have potential for predictive modelling 12
Fault/warning messages/day
Qantas A380 Fleet
How to identify the right messages to focus limited resources and reduce costly downtime?
Sample Existing Predictive Maintenance Journey
Develop ML model (MATLAB)
alongside local university
Optimise code Reduce runtime
Develop user web front end
Build evaluation module
Refine model parameters
Years
Microsoft Azure ML Predictive Maintenance Journey
Configure model in AML PM template
Evaluate & refine model data & parameters
Visualize results in Power BI
Months
Orchestrate data pipeline in Azure Data Factory

6. Aka.ms/StrataPDM

7. Ready for ML approach? Question is sharp.
5/6/ :10 AM
Ready for ML approach?
The better the raw materials, the better the product.
Question is sharp.
Data measures what they care about.
Data is accurate.
Data is connected.
A lot of data.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
E.g. Predict whether component X will fail in the next Y days; clear path of action with answer
E.g. Identifiers at the level they are predicting
E.g. Failures are really failures, human labels on root causes; domain knowledge translated into process
E.g. Machine information linkable to usage information
E.g. Will be difficult to predict failure accurately with few examples

8. Data Sources FAILURE HISTORY REPAIR HISTORY MACHINE CONDITIONS
5/6/ :10 AM
Data Sources
FAILURE HISTORY
REPAIR HISTORY
MACHINE CONDITIONS
The failure history of a machine or component within the machine.
The repair history of a machine, e.g. previous maintenance records, components replaced, maintenance activities performed.
The operating characteristics of a machine, e.g. data collected from sensors.
MACHINE FEATURES
OPERATING CONDITIONS
OPERATOR ATTRIBUTES
The features of machine or components, e.g. production date, technical specifications.
Environmental features that may influence a machine’s performance, e.g. location, temperature, other interactions.
The attributes of the operator who uses the machine, e.g. driver.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Feature Engineering Better features are better than better algorithms…
5/6/ :10 AM
Feature Engineering
Better features are better than better algorithms…
Rolling Aggregates
Tumbling Aggregates
Static Features
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
E.g. Mean, Min, Max over the last 3 hours
E.g. Mean, Min, Max for every hour in the last 3 hours
E.g. Years in service, model

10. BINARY CLASSIFICATION MULTICLASS CLASSIFICATION
Modelling Techniques
BINARY CLASSIFICATION
REGRESSION
Predict failures within a future period of time
E.g. Is the engine going to fail in the next 24 hours?
Predict remaining useful life, the amount of time before the next failure
E.g. How long will an aircraft engine last before it fails?
MULTICLASS CLASSIFICATION
ANOMALY DETECTION
Predict failures with their causes within a future time period.
Predict remaining useful life within ranges of future periods
Identify change in normal trends to find anomalies

11. Tips & Tricks How to split into training and validation sets
5/6/ :10 AM
Tips & Tricks
How to split into training and validation sets
Be careful of “leakage”
Best practice to consider : time based split etc.
Imbalanced Data
Cost-sensitive learning
Sampling methodologies
Report appropriate metrics
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. How do you know if the model is good? 99% accuracy? Is 0.2 recall good?

13. Establishing Baseline Metrics for Classification Models
Building trivial classifiers – 4 different ways
Random Guess
Majority Class
Weighted Guess by distribution
Weighted Guess by decision threshold
Compute Metrics – Accuracy, Precision, Recall
Examples

14. Model Evaluation 5/6/2018 11:10 AM
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Ground Truth 𝑃𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑆𝑖𝑧𝑒 =𝑛
𝑃𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑆𝑖𝑧𝑒 =𝑛
𝐹𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑙𝑎𝑏𝑒𝑙𝑙𝑒𝑑 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 =𝑥
𝐹𝑟𝑎𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑙𝑎𝑏𝑒𝑙𝑙𝑒𝑑 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 = 1−𝑥
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑙𝑎𝑏𝑒𝑙𝑙𝑒𝑑 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃 =𝑥𝑛
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑛𝑠𝑡𝑎𝑛𝑐𝑒𝑠 𝑙𝑎𝑏𝑒𝑙𝑙𝑒𝑑 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑁 = 1−𝑥 𝑛
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦= 𝑇𝑃+𝑇𝑁 𝑛
𝑅𝑒𝑐𝑎𝑙𝑙= 𝑇𝑃 𝑇𝑃+𝐹𝑁
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= 𝑇𝑃 𝑇𝑃+𝐹𝑃
𝑇𝑃𝑅= 𝑇𝑃 𝑃 , 𝐹𝑃𝑅= 𝐹𝑃 𝑁
For derivation please refer to the paper here.

16. Three steps from baseline..
Step 1 : Express assumptions mathematically
Step 2: Create a Confusion Matrix
Step 3: Compute Baseline Metrics

17. Machine Learning, Analytics, & Data Science Conference
Random Guess
5/6/ :10 AM
randomly assign half of the labels to P and the other half as N
𝑇𝑃+𝐹𝑃= 𝑛 𝑇𝑁+𝐹𝑁= 𝑛 2
𝑇𝑃=𝐹𝑁, 𝐹𝑃=𝑇𝑁
𝑻𝑷, 𝑻𝑵, 𝑭𝑷,𝑭𝑵 −𝑪𝒐𝒏𝒇𝒖𝒔𝒊𝒐𝒏 𝑴𝒂𝒕𝒓𝒊𝒙
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦
𝑅𝑒𝑐𝑎𝑙𝑙𝑖
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑖
(𝑇𝑃𝑅 , 𝐹𝑃𝑅)
Binary Classification
0.5 𝑥
(0.5 , 0.5)
Multiclass Classification
1 𝑘
𝑥 𝑖
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
𝑥 𝑖 is fraction of instances belonging to class 𝑖 (𝑖 = 1 to 𝑘)
𝑘 = No. of classes

18. Machine Learning, Analytics, & Data Science Conference
5/6/ :10 AM
Majority Class
assigning all the labels to negative assuming that is the majority class.
𝑇𝑁+𝐹𝑁=𝑛
𝑇𝑃=0
𝐹𝑃=0
𝑻𝑷, 𝑻𝑵, 𝑭𝑷,𝑭𝑵 −𝑪𝒐𝒏𝒇𝒖𝒔𝒊𝒐𝒏 𝑴𝒂𝒕𝒓𝒊𝒙
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦
𝑅𝑒𝑐𝑎𝑙𝑙𝑖
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑖
(𝑇𝑃𝑅 , 𝐹𝑃𝑅)
Binary Classification
(1−𝑥)
𝑁𝑎𝑁
(0,0)
Multiclass Classification
1−𝑥 𝑖=𝑚
1𝑖𝑚
0𝑖𝑚
𝑥 𝑖=𝑚
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
𝑥 𝑖=𝑚 is fraction of instances belonging to majority class (negative class is majority class here)
𝑥 𝑖 is fraction of instances belonging to class 𝑖 (𝑖 = 1 to 𝑘)
𝑘 = No. of classes

19. Weighted Guess by Distribution
Machine Learning, Analytics, & Data Science Conference
Weighted Guess by Distribution
5/6/ :10 AM
x% of actual positives are assigned as positive by this model and
(1-x) % of actual negatives are assigned as negatives
𝑇𝑃=𝑥𝑃 𝑇𝑁= 1−𝑥 𝑁
𝑻𝑷, 𝑻𝑵, 𝑭𝑷,𝑭𝑵 −𝑪𝒐𝒏𝒇𝒖𝒔𝒊𝒐𝒏 𝑴𝒂𝒕𝒓𝒊𝒙
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦
𝑅𝑒𝑐𝑎𝑙𝑙𝑖
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑖
(𝑇𝑃𝑅 , 𝐹𝑃𝑅)
Binary Classification
𝑥 2 + (1−𝑥) 2 𝑥
(𝑥,𝑥)
Multiclass Classification
𝑖=1 𝑘 𝑥 𝑖 2
𝑥 𝑖
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
𝑥 𝑖 is fraction of instances belonging to class 𝑖 (𝑖 = 1 to 𝑘)
𝑘 = No. of classes

20. Weighted Guess by Threshold
Machine Learning, Analytics, & Data Science Conference
5/6/ :10 AM
(1-t)% of actual positives (P) are assigned as positive by this model
and t% of actual negatives (N) are assigned as negatives
𝑇𝑃=(1−𝑡)𝑃
𝑇𝑁=𝑡𝑁
𝑻𝑷, 𝑻𝑵, 𝑭𝑷,𝑭𝑵 −𝑪𝒐𝒏𝒇𝒖𝒔𝒊𝒐𝒏 𝑴𝒂𝒕𝒓𝒊𝒙
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦
𝑅𝑒𝑐𝑎𝑙𝑙𝑖
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑖
(𝑇𝑃𝑅 , 𝐹𝑃𝑅)
Binary Classification
𝑃+𝑡(𝑁−𝑃) 𝑛
(1−𝑡)
𝑃 𝑛
(1−𝑡,1−𝑡)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Predict machine failing …
Red = Actual failure points (label = 1)
Blue = Non-failure points
(label = 0)
Threshold of 0.23
Window of 5 days for labeling
Example: Should be predicted to be failure, but given 0.23 threshold, predicted as 0 and thus false negative
Failure point

22. Machine Learning, Analytics, & Data Science Conference
Custom R module
5/6/ :10 AM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Machine Learning, Analytics, & Data Science Conference
Custom R module
Machine Learning, Analytics, & Data Science Conference
5/6/ :10 AM
Full code in Github as well
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. The Cost Factor Classification – Cost sensitive methods
Classification – Cost in-sensitive methods, manipulate the predictions
FP Error Rate = FP/n
FN Error Rate = FN/n
Error Rate = (FP + FN) / n
FP Cost = FP Error Rate * $ cost of FP
FN Cost = FN Error rate * $ cost of FN
Cost = FP Cost + FN Cost

25. Predictive Maintenance Scenarios
Manufacturing Line
less than 1% of failure data available
tolerant of false positive even at the expense of false negatives
tuned the model parameters for high AUC
chose a high recall ROC operation point - caught failures 75% of the time
Wind farm
only 1% of data constituting failure
tolerant of a false negative over a false positive
tuned the model parameters for high F1 score.
F1 score = 0.07 ! Still 3 X what a random model would have produced (0.02)

26. Predict machine failing …
Red = Actual failure points (label = 1)
Blue = Non-failure points
(label = 0)
Threshold of 0.23
Window of 5 days for labeling
Example: Should be predicted to be failure, but given 0.23 threshold, predicted as 0 and thus false negative
Failure point

27. Think about cost/benefit ratio and remember than “99% accuracy” means nothing without context!

28. Learn and try yourself! Learn from Cortana Intelligence Gallery
5/6/2018
Learn and try yourself!
Learn from Cortana Intelligence Gallery
Solution package material – deploy by hand to learn here
Try Cortana Intelligence Solution Template – Predictive Maintenance for Aerospace
Try Azure IOT pre-configured solution for Predictive Maintenance
Read the Predictive Maintenance Playbook for more details on how to approach these problems
Run the Modelling Guide R Notebook for a DS walk-through
Baseline Metrics overview
Performance Metrics overview
For more details on metrics: Blogs and paper
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.