Visualizing Complex Software Systems Wim De Pauw IBM T.J. Watson Research Center wim@us.ibm.com VAC Consortium Meeting July 31, 2012

Outline Software Visualization is Big Data: dynamic analysis of complex systems debugging, performance analysis, understanding Demos: Distributed computing Streaming applications Cloud Watson DeepQA Log analysis: Cisco, finance, HDFS Conclusion

Big Data Moving target Sales data, web traffic, social media, sensors Not just sheer numbers: Multi-dimensional Inter-correlated High speed Transient Different degrees of relevance Software visualization: understanding the behavior of complex systems from Big Trace Data

Big Data = Smart Data ? Enormous amounts of data are collected What is the cost to collect and process data? Need for data specialists Are we collecting the right data in order to: Draw the right conclusions Discover the unexpected See the relevant trends Find outliers Help with troubleshooting Alert

Interaction with various layers to handle complexity Interaction only with the visualization layer. Changes the view, not the underlying elements visualization modeling E.g. drill down, select. Interaction with visualization and slightly with models visualization modeling The user can navigate through the results of the modeling “User type” is somewhat correlated with interaction, but not 100% Collaboration/Communication somewhat related to user type Strong interaction with visualization models, analytics, and/or data collection visualization analytics User can perform pattern extraction, online analysis; event collection is adaptive modeling data collection

Visualizing Big Data visualization Strong interaction with visualization, models, analytics, and/or data collection analytics modeling data collection Making data visible, tactile, easy to manipulate: Enable and discover insight: illuminating the unexpected Work with higher level concepts: patterns, transactions, outliers

Visualize the mental model Design crisp abstractions: show concepts specific for user-role horizontal layered approach Become domain expert Bridge gap between static and dynamic world Tool is a collaboration medium Example: Web Services Navigator

Iterative design cycle Work with users early on understand the problem solving methodology in domain iterative collaboration using incremental prototypes Use common theme, different facets Best if visualization is part of the emerging platform Demo: Streamsight

Visualization shows the unexpected Leave room for discovery Impossible to foresee or automate all tasks demo: Imagesight (cloud computing)

Combining analysis and interactive visualization Demo: Virtual images: content analysis and consolidation Watson DeepQA: feature analysis

Virtual Images: Content Analysis and Consolidation

Watson DeepQA: Feature Analysis

Analyzing Systems for Big Data Demo: Watson Job scheduler Classic Stacked Graph Floating bundles

Job Scheduling for Watson

Analyzing Big Log Data Demo: Cisco logs: discrete events ACME: complex Service Oriented Architecture HDFS: visualizing data transfers in MapReduce

Cisco log

Hadoop logs: HDFS & MR Shuffles

With contributions from many colleagues Conclusion Managing, analyzing, optimizing and troubleshooting complex software systems requires: Integration of visualization, analytics, modeling, data acquisition and interaction Leveraging the rich inherent structures of the domain Providing actionable, (pseudo-)live systems With contributions from many colleagues