Ultra-Scale Visualization with Open-Source Software Berk Geveci Kitware Inc.

Background

Kitware Is A software company creating open-source collaboration platforms which are used globally for research teaching commercial application. This software is created by internationally recognized experts in extended communities using a rigorous, quality-inducing software development process.

Technical Portfolio

Software Process

Heart and Soul: Software Quality Process

In The Beginning There Was VTK

From Ohloh: Very large, active development team: Over the past twelve months, 66 developers contributed new code to VTK. This is one of the largest open-source teams in the world, and is in the top 2% of all project teams on Ohloh. VTK Development Team and many others...

VTK Development Team

Then Came ParaView

ParaView Architecture

Support for Large Displays

ParaView is VTK ParaView is developed by the VTK team The ParaView team develops in the VTK repository : all development benefits the whole community ParaView leverages all features of VTK

ParaView is a Framework ParaView extends VTK to provide: Client-server computing State management Python modules Application/GUI framework ParaView framework can be used to develop other applications ParaView can be embedded in other application and frameworks

Short Demo

VTK and ParaView Funding Army Aeroflightdnymics Directorate National Alliance for Medical Computing and many others...

Problem Statement

The Good

Ability to Compute

Ability to Communicate

Remote Access

The Bad Ability to Compute >> Ability to Transfer >> Ability to Store/Read

State of Large Data Vis

Parallel Visualization

Visualization Resources LANL Vis Cluster (2001): 128 dual-core nodes with NVIDIA Quadro FX 540 Sandia Red Rose (2005): 264 dual-core nodes with NVIDIA Quadro FX 3400 TACC Longhorn (2009): core nodes with 2 NVIDIA Quadro FX 5800s

General Purpose Tools EnSight ParaView VisIt FieldView …

Specialized Tools VAPOR ViSUS...

1 billion cell asteroid detonation simulation (AMR) ½ billion element weather simulation (rectilinear) 150 million element fire simulation (unstructured)

Where to Next?

Large Data, Small Bandwidth and Small Vis Resources Vis on the supercomputer Batch Interactive Co-Processing Multi-Resolution Streaming

Ability to Compute

Vis on Supercomputer

Compiling Parallel IO Scalability Software Rendering Compositing Client/server Resource allocation

Client/server Resource Allocation Interactive on Supercomputer

ParaView on Supercomputers Blue Gene L and P Cray Xt 3, 4 and 5 AIX Linux Visualization of VPIC results on Kraken (image courtesy of Bill Daughton, LANL)

Co-Processing

11

Run-Time Visualization and Setup

Objectives Develop an extensible and flexible co- processing library Develop CFD specific analysis and visualization algorithms Develop tools for visualizing, analyzing and managing extracts Integrate run-time visualization with co- processing

Workflow

Setup Use ParaView Load and visualize extracts or previous simulation output for context Interactively build processing/visualization pipeline using available algorithms Save configuration file for co- processing library

Library Architecture Based on VTK/ParaView C, Fortran and Python bindings Built-in Python interpreter (optional) Full demand-driven pipeline Distributed computing with MPI Extract generation Rendering (off-screen with Mesa)

Connection with the Simulation simulation_initialize() coprocessor_initialize() for t in time_steps: do_compute() if coprocess_needed(): do_coprocess() coprocessor_finalize() simulation_finalize()

Connection with the Simulation Adapter or converter? Simple memory layout (structured data) We used converters for vectors, adapter for scalars Can we use adapters for both? Complex memory layout (unstructured grid connectivity) We used converters Possible to use generic dataset framework

Decision Making Based on Features Steer processing based on emerging features For example: Change extraction parameters Extract more or less often Demand-driven pipeline help Python bindings are ideal

Simulations – the More the Merrier Overflow (overset curvilinear) Phasta (unstructured) Acusolve (unstructured) Helios (unstructured - AMR hybrid) CTH (AMR) NPIC (structured)...

Run-Time Visualization and Setup

What Is Different? Extensible and flexible VTK API Python Scalable Leverages a larger system (ParaView) Tools for visualizing, analyzing and managing extracts

Management and Visualization of Extracts Extracts are saved multi- resolution Viewer streams extracts based on camera position Extracts are registered with a database (with meta-data) for data management

Multi-Resolution Streaming

Data-Parallel Processing

Streaming

Prioritized Streaming

Adaptive Streaming

Collaboration and Web Visualization

Remote Access

Collaboration

Web Visualization

Architecture

Comparative Visualization

Engine inlet with aggressive bend Active control Parameters: duct geometry, location of jet, gap width, jet waveform, jet frequency,...

Ouch!

Data and Meta-Data Management

Batch Analysis

Loading Meta-Data in ParaView

Analysis and Subsetting of Meta- Data

Aperture for Data- Analysis

Conclusions

Compute power is growing fast Network speed is growing fast Researchers can simulate more Visualization resources are not scaling (budget) It is about insight Multiple solutions: In-situ Multi-resolution streaming Vis on supercomputer It is not always about size: Collaboration Ensemble analysis Need customized solutions Need frameworks to make it easy to build them Multiple slides from these

Ability to Compute and Communicate

Bottlenecks

We need multiple solutions!

Interactive on Supercomputer

Co-Processing

Multi-Resolution Streaming

It’s About Collaboration

We need frameworks to build specialized tools

General Purpose Tools EnSight ParaView VisIt FieldView …

Specialized Tools VAPOR ViSUS...

ParaView is a Framework ParaView is VTK

The End