1. Visualizing Complex Software Systems
Wim De Pauw
IBM T.J. Watson Research Center
VAC Consortium Meeting
July 31, 2012

2. Outline Software Visualization is Big Data:
dynamic analysis of complex systems
debugging, performance analysis, understanding
Demos:
Distributed computing
Streaming applications
Cloud
Watson DeepQA
Log analysis: Cisco, finance, HDFS
Conclusion

3. Big Data Moving target Sales data, web traffic, social media, sensors
Not just sheer numbers:
Multi-dimensional
Inter-correlated
High speed
Transient
Different degrees of relevance
Software visualization: understanding the behavior of complex systems from Big Trace Data

4. Big Data = Smart Data ? Enormous amounts of data are collected
What is the cost to collect and process data?
Need for data specialists
Are we collecting the right data in order to:
Draw the right conclusions
Discover the unexpected
See the relevant trends
Find outliers
Help with troubleshooting
Alert

5. Interaction with various layers to handle complexity
Interaction only with the visualization layer. Changes the view, not the underlying elements
visualization
modeling
E.g. drill down, select.
Interaction with visualization and slightly with models
visualization
modeling
The user can navigate through the results of the modeling
“User type” is somewhat correlated with interaction, but not 100%
Collaboration/Communication somewhat related to user type
Strong interaction with visualization models, analytics, and/or data collection
visualization
analytics
User can perform pattern extraction, online analysis;
event collection is adaptive
modeling
data collection

6. Visualizing Big Data
visualization
Strong interaction with visualization, models, analytics, and/or data collection
analytics
modeling
data collection
Making data visible, tactile, easy to manipulate:
Enable and discover insight: illuminating the unexpected
Work with higher level concepts: patterns, transactions, outliers

7. Visualize the mental model
Design crisp abstractions:
show concepts specific for user-role
horizontal layered approach
Become domain expert
Bridge gap
between static and dynamic world
Tool is a collaboration medium
Example: Web Services Navigator

8. Iterative design cycle
Work with users early on
understand the problem solving methodology in domain
iterative collaboration using incremental prototypes
Use common theme, different facets
Best if visualization is part of the emerging platform
Demo: Streamsight

9. Visualization shows the unexpected
Leave room for discovery
Impossible to foresee or automate all tasks
demo: Imagesight (cloud computing)

10. Combining analysis and interactive visualization
Demo:
Virtual images: content analysis and consolidation
Watson DeepQA: feature analysis

11. Virtual Images: Content Analysis and Consolidation

12. Watson DeepQA: Feature Analysis

13. Analyzing Systems for Big Data
Demo: Watson Job scheduler
Classic Stacked Graph
Floating bundles

14. Job Scheduling for Watson

15. Analyzing Big Log Data Demo: Cisco logs: discrete events
ACME: complex Service Oriented Architecture
HDFS: visualizing data transfers in MapReduce

16. Cisco log

17. Hadoop logs: HDFS & MR Shuffles

18. With contributions from many colleagues
Conclusion
Managing, analyzing, optimizing and troubleshooting complex software systems requires:
Integration of visualization, analytics, modeling, data acquisition and interaction
Leveraging the rich inherent structures of the domain
Providing actionable, (pseudo-)live systems
With contributions from many colleagues