FINDING PATTERNS IN TEMPORAL DATA KRIST WONGSUPHASAWAT TAOWEI DAVID WANG CATHERINE PLAISANT BEN SHNEIDERMAN HUMAN-COMPUTER INTERACTION LAB UNIVERSITY OF MARYLAND 27th HCIL Symposium May 27, 2010

FINDING PATTERNS IN TEMPORAL DATA KRIST WONGSUPHASAWAT TAOWEI DAVID WANG CATHERINE PLAISANT BEN SHNEIDERMAN HUMAN-COMPUTER INTERACTION LAB UNIVERSITY OF MARYLAND 27th HCIL Symposium May 27, 2010

Patient ID: /02/ :26Arrival 12/02/ :36Emergency 12/02/ :44ICU 12/05/ :07Floor 12/14/ :19Exit Time Emergency ICU Floor Exit TEMPORAL CATEGORICAL DATA A type of time series 04/26/ : /26/ : /26/ : /26/ : /26/ : Event Category Stock: Microsoft Numerical Arrival Event

TEMPORAL CATEGORICAL DATA Electronic Health Records: symptoms, treatment, lab test Traffic incident logs: arrival/departure time of each unit Student records: course, paper, proposal, defense, etc. Others: web logs, usability study logs, etc.

10+ years work on temporal visualization (mostly on Electronic Health Records)

LIFELINES SINGLE RECORD [Plaisant et al. 1998]

LifeLines – Single Patient

working with physicians at WASHINGTON HOSPITAL CENTER

EXAMPLE DATA Patient transfers ARRIVALArrive the hospital EMERGENCYEmergency room ICUIntensive Care Unit INTERMEDIATEIntermediate Medical Care FLOORNormal room EXIT-ALIVELeave the hospital alive EXIT-DEADLeave the hospital dead

TASKS within 2 days ICUFloorICU Example: Finding “Bounce backs”

LIFELINES 2 RECORD [Wang et al. 2008, 2009]

LifeLines2 – Search and Visualize ARF (Align-Rank-Filter) Framework Temporal Summary Multiple Records

ALIGNMENT Sentinel events as reference points Time Patient # Arrival Emergency ICU Floor Exit Patient # Arrival Emergency ICU Floor Exit JuneJuly August

ALIGNMENT (2) Time shifting Time Patient # Admit Emergency ICU Floor Exit Patient # Admit Emergency ICU Floor Exit 01 M 2 M

SIMILAN RECORD [Wongsuphasawat & Shneiderman 2009]

Similan – Search by Similarity

FINDING “BOUNCE BACKS” BeforeAfter Much faster to specify new query Visualizing the results gives better understanding

USER STUDIES: SEARCH Exact MUST have A, B, C Record#1 Record#2 Record#3 more similar Similarity-based SHOULD have A, B, C SimilanLifeLines2 Query Record#2 Record#1 Record#3 Query

USER STUDIES: SEARCH Exact MUST have A, B, C Similarity-based SHOULD have A, B, C Query Record#1 Record#2 Record#3 more similar Query Record#2 Record#1 Record#3 SimilanLifeLines2 1

NEW STUFF Needs for an overview -> LifeFlow!

TASKS within 2 days ICUFloorICU Example: Finding “Bounce backs” Other questions Arrival ICU ? ? ?

LIFEFLOW RECORD AGGREGATE Merge multiple records into tree VISUALIZE Display the aggregation

AGGREGATE Aggregate by prefix #1 #2 #3 #4 Example with 4 records

AGGREGATE Aggregate by prefix #1 #2 #3 #4

VISUALIZE Inspired by the Icicle tree [Fekete 2004] Number of files

VISUALIZE (2) Use horizontal axis to represent time Video

DEMO – LIFEFLOW When the lines are combined into flow

FUTURE WORK Comparison Jan-Mar 2008April-June 2008 Intermediate ICU IntermediateICU Floor

TAKE-AWAY MESSAGE Information visualization is a powerful way to explore temporal patterns. You can work with us on new case studies.

TEMPORAL CATEGORICAL DATA Electronic Health Records: symptoms, treatment, lab test Traffic incident logs: arrival/departure time of each unit Student records: course, paper, proposal, defense, etc. Others: web logs, usability study logs, etc.

EXAMPLE – TRAFFIC INCIDENTS

ACKNOWLEDGEMENT D R. P HUONG H O, D R. M ARK S MITH, D AVID R OSEMAN W ASHINGTON H OSPITAL C ENTER N ATIONAL I NSTITUTES OF H EALTH (NIH) - G RANT CA M ICHAEL P ACK, M ICHAEL V AN D ANIKER C ENTER FOR A DVANCED T RANPORTATION T ECHNOLOGY L AB (CATT L AB )

TAKE-AWAY MESSAGE Information visualization is a powerful way to explore temporal patterns. You can work with us on new case studies. More demos this afternoon {kristw, tw7, plaisant,

Q&A Questions? {kristw, tw7, plaisant,

THANK YOU Thank you

BACKUP SLIDES Junkyard...

LIFELINES2 8 case studies –Bounce backs –Step ups –BIPAP –Etc.

DR. P

Does not help exploring sequential patterns Needs a new overview LifeLines2’s Temporal Summary [Wang et al. 2009] Continuum’s Histogram [Andre 2007]

USER STUDIES 8 Extensive case studies Compared LifeLines2 with Similan –Learn advantages & disadvantages Drawing is preferred Clear cut off points is needed Working on improvements –Flexible temporal search

SIMILAN Compared with LifeLines2 in an experiment –Learn advantages & disadvantages –Drawing is preferred –No clear cut off points Working on improvements –Flexible temporal search

LIFEFLOW AGGREGATE VISUALIZE Merge multiple records into tree Display the tree

APPROACHES Exact Search MUST have A, B, C Similarity-based Search SHOULD have A, B, C Query Record#1 Record#2 Record#3 more similar Query Record#1 Record#2 Record#3

MOTIVATION RESEARCH QUESTIONS RESEARCH QUESTION#1 PRELIM. + PROPOSED WORK CONCLUSION RESEARCH QUESTION#2 PRELIM. + PROPOSED WORK

EXPECTED CONTRIBUTIONS 1.Design of visual representations, user interfaces and interaction techniques 2.Algorithms for flexible temporal search 3.Evaluation results 4.Open new directions for exploring temporal categorical data

NEEDS FOR AN OVERVIEW We learn

NEEDS Visualize overview or show summary Where should I start?

TEMPORAL VISUALIZATIONS Background and related work

RELATED WORK Single record Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Visualization E.g. LifeLines, MIDGAARD, etc.

RELATED WORK (2) Multiple records Visualization E.g. LifeLines2, Continuum, ActiviTree, etc. Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit

More space please....

INFORMATION VISUALIZATION MANTRA OVERVIEW FIRST, ZOOM AND FILTER, THEN DETAILS ON DEMAND

RELATED WORK (3) Multiple records Visualization E.g. LifeLines2, Continuum Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit Patient ID: /02/ :26Arrival 12/02/ :26Emergency 12/02/ :44ICU 12/05/ :07Floor 12/08/ :02Floor 12/14/ :19Exit

SEQUENTIAL PATTERNS within 2 days ICUFloorICU Examples: “Bounce backs” Patient #1 Patient #2 Patient #3 Patient #4

DESIGN AN OVERVIEW Sequential patterns Scalability vs. Loss of information