Uncertainty in Automation: Anomaly Detection in Event-Based Systems Dawn Tilbury Linday Allen (PhD) and John Broderick University of Michigan

2 Outline Example problem: inconsistent logical behavior Developed solution: Anomaly detection –Model generation using observed data –Performance assessment of models using known “good” and “bad” behavior –On-line anomaly detection Industrial Application –Academic assumptions meet industry realities –Resolution and results Lessons learned

3 Example from our testbed s1 Part1 PartReady Release Pallet LoadPart1 Part1 PartReady Correct, typical behaviorIncorrect behavior No model of entire system’s correct behavior Manual inspection required to find this anomaly Laborious, offline ResponseOPC Tag ResponseOPC Tag

4 Approach: Anomaly detection using model generation Goal: diagnosis of system level event- based faults in mfg systems without use of pre-existing formal model Method:  Generate models based on training data  Detect anomalies on-line by comparing traces to models  Advise the operator when anomaly occurs Plant Logic Control System Fault diagnosis Anomaly detection Operator Controller All Events Events Op. Feedback Anomalies

Knowns & Unknowns Known: –Resources in system Robots CNC machines Pallets Measurable: –OPC tag changes Communication events between controllers Unknown: –Formal model of the system Could be constructed but is time-consuming and error prone –Logic control code Written by different people at different times in different languages –Correct event order Many different orders may be acceptable

6 Solution approach Model Generation adbe... efad... abfc... ok Resource Info eafd... Performance Assessment acea... ok not Fault Detection abfe... ok abfe... not OR Given resource information and strings of “ok” events Create a set of models that can generate these strings Given some “ok” and “not ok” strings, compute the performance of each model Given a new string, determine whether the models accept it (weight by model performance) If not, where is the anomaly

Anomaly Detection Method Inputs: –Streams of events from system in operation –Resource information, including mapping of events to resources Outputs: –Set of models that represent the system behavior –Model performance on training data On-line detection: –Score for each string – anomalous or not 7

Machining Cell: Physical Set-Up Problem: G2 will have raw parts and at least one CNC available, but G2 incorrectly waits Resources: –Gantries, CNCs, buffer at hand-off Events: PLC data recorded via Ford data collection system G1G2M1M EntryHand-off Reject Exit

Data collection set-up Data from each machine & gantry –Bits include: Cycle End, Good/Bad Cycle, Wait Aux, Blocked, and Starved –PLC message generated each time particular bit changes occur Approx. 11,000 parts worth of data 9 IT System PLC Driving Logic Driving Logic Function Block PLC Driving logic Function Block Driving Logic (270,000 PLC messages)

Identified Inconsistencies What we thought we would get: –Well-defined strings of events –Events that acquire/release resources recorded –Unique mapping of PLC bits to events –Many strings, starting from the initial state, labeled as “good” or “bad What we got: –Not every event triggers a message  multi-bit change (order is uncertain) –Not all resource events captured in data collection –Some bits used for multiple purposes –One huge log file with no defined “beginning”

Resolution of Inconsistencies Academic Assumptions Industry RealitiesResolution 1 Resource events available Some events filtered in data collection I: Logic changed 2 String of ordered events Multiple bit changes per message possible A: Heuristic decision algorithm 3 Consistent bit- meaning mapping Inconsistent bit- meaning mapping I, A: Logic changed, pre-process data 4 Event streams start in initial state System runs continuously A: Nec. condition to create stream 5 Separate, labeled streams Continuous, unlabeled stream A: Splitting, labeling algorithm

Ford data Word 18 bits 8-10 give the CNC ID Gantry waiting: word 19 bit 9 is high

Lessons learned Sometimes you can adapt/improve your method to handle given uncertainties –Multiple models when system model unknown –Multiple bit changes  uncertain event order –Initial state unknown –Advise operator instead of closing the loop Sometimes you have to decrease the uncertainty by improving the system –Consistent bit/event mapping –Unobservable events for acquiring resources 13

Future work 14