Large Scale Data Visualization with VisIt Hank Childs, Lawrence Berkeley, August 8, 2012 Basic usage Data analysis Derived quantities Scripting Moviemaking.

Slides:



Advertisements
Similar presentations
Operating-System Structures
Advertisements

EUFORIA FP7-INFRASTRUCTURES , Grant JRA4 Overview and plans M. Haefele, E. Sonnendrücker Euforia kick-off meeting 22 January 2008 Gothenburg.
1 Slides presented by Hank Childs at the VACET/SDM workshop at the SDM Center All-Hands Meeting. November 26, 2007 Snoqualmie, Wa Work performed under.
ParaView Tutorial Greg Johnson, Karla Vega. Before we begin… Make sure you have ParaView installed so you can follow along in the lab section –
Scientific Visualization with VisIt
Petascale I/O Impacts on Visualization Hank Childs Lawrence Berkeley National Laboratory & UC Davis March 24, B element Rayleigh-Taylor Instability.
McIDAS-V McIDAS-V The 5 th Generation of McIDAS by Tom Whittaker Space Science and Engineering Center University of Wisconsin-Madison USA with contributions.
DEV392: Extending SharePoint Products And Technologies Through Web Parts And ASP.NET Clint Covington, Program Manager Data And Developer Services - Office.
Chandler ISR June Chandler Open Source Personal Information Manager , calendar, contacts, tasks, free-form items Easy sharing and collaboration.
VisIt Software Engineering Infrastructure and Release Process LLNL-PRES Lawrence Livermore National Laboratory, P. O. Box 808, Livermore,
Lawrence Livermore National Laboratory Visualization and Analysis Activities May 19, 2009 Hank Childs VisIt Architect Performance Measures x.x, x.x, and.
Scientific Visualization with VisIt Eliot Feibush Bum Shik Kim Zach Kaplan PRINCETON PLASMA PHYSICS LABORATORY PICSciE Princeton Institute for Computational.
Cambodia-India Entrepreneurship Development Centre - : :.... :-:-
Challenges and Solutions for Visual Data Analysis on Current and Emerging HPC Platforms Wes Bethel & Hank Childs, Lawrence Berkeley Lab July 20, 2011.
Virtual Machine Management
Talend 5.4 Architecture Adam Pemble Talend Professional Services.
Automation using Selenium Authored & Presented by : Chinmay Sathe & Amit Prabhu Cybage Software Pvt. Ltd.
Project Proposal: Academic Job Market and Application Tracker Website Project designed by: Cengiz Gunay Client: Cengiz Gunay Audience: PhD candidates and.
NA-MIC National Alliance for Medical Image Computing Core 1b – Engineering End-user Platform Steve Pieper Isomics, Inc.
Linux Operations and Administration
Leicester, February 24, 2005 VisIVO, a VO-Enabled tool for Scientific Visualization and Data Analysis. VO-TECH Project. Stage01 Ugo Becciani INAF – Astrophysical.
CS110/CS119 Introduction to Computing (Java)
11 4 VisIt is a richly featured, turnkey application VisIt is an open source, end user visualization and analysis tool for simulated and experimental data.
Scientific Visualization with VisIt Eliot Feibush PLASMA PRINCETON PHYSICS LABORATORY PICSciE Princeton Institute for Computational Science and Engineering.
VisIt: a visualization tool for large turbulence simulations  Outline Success stories with turbulent simulations Overview of VisIt project 1 Hank Childs.
SOFTWARE ENGINEERING1 Introduction. Software Software (IEEE): collection of programs, procedures, rules, and associated documentation and data SOFTWARE.
1 Apache. 2 Module - Apache ♦ Overview This module focuses on configuring and customizing Apache web server. Apache is a commonly used Hypertext Transfer.
Introduction With TimeCard users can tag SharePoint events with information that converts them into time sheets. This way they can report.
IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Collaborating with iPlant.
The Cluster Computing Project Robert L. Tureman Paul D. Camp Community College.
VACET: Deploying Technology for Visualizing and Analyzing Astrophysics Simulations Author May 19, 2009.
Nov. 14, 2012 Hank Childs, Lawrence Berkeley Jeremy Meredith, Oak Ridge Pat McCormick, Los Alamos Chris Sewell, Los Alamos Ken Moreland, Sandia Panel at.
1 Computer Programming (ECGD2102 ) Using MATLAB Instructor: Eng. Eman Al.Swaity Lecture (1): Introduction.
Contents 1.Introduction, architecture 2.Live demonstration 3.Extensibility.
IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Collaborating with iPlant.
Ch 1. A Python Q&A Session Spring Why do people use Python? Software quality Developer productivity Program portability Support libraries Component.
Loosely Coupled Parallelism: Clusters. Context We have studied older archictures for loosely coupled parallelism, such as mesh’s, hypercubes etc, which.
AVS/Express and VisIt Training NERSC Users meeting June PPPL Cristina Siegerist NERSC/LBNL Visualization group June 13, 2006.
Efficient Visualization and Analysis of Very Large Climate Data Hank Childs, Lawrence Berkeley National Laboratory December 8, 2011 Lawrence Livermore.
1 CSE 390 Lecture 9 Do anything from anywhere – tools to free your choice of OS slides created by Marty Stepp, modified by Jessica Miller and Ruth Anderson.
Graphing and statistics with Cacti AfNOG 11, Kigali/Rwanda.
LLNL-PRES This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344.
240-Current Research Easily Extensible Systems, Octave, Input Formats, SOA.
CS4710 Why Progam?. Why learn to program? Utility of programming skills: understand tools modify tools create your own automate repetitive tasks automate.
VisIt is an open source, richly featured, turn-key application for large data.  Used by:  Visualization experts  Simulation code developers  Simulation.
LBNL VACET Activities Hank Childs Computer Systems Engineer - Visualization Group August 24, 2009.
Esri UC2013. Technical Workshop. Technical Workshop 2013 Esri International User Conference July 8–12, 2013 | San Diego, California Sharing Workflows with.
Hank Childs, University of Oregon Volume Rendering Primer / Intro to VisIt.
Visualization with ParaView. Before we begin… Make sure you have ParaView 3.14 installed so you can follow along in the lab section –
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
EMI INFSO-RI ARC tools for revision and nightly functional tests Jozef Cernak, Marek Kocan, Eva Cernakova (P. J. Safarik University in Kosice, Kosice,
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
1 27B element Rayleigh-Taylor Instability (MIRANDA, BG/L) VisIt: a visualization tool for large turbulence simulations Large data requires special techniques.
Clusters Rule! (SMPs DRUEL!) David R. White Sandia National Labs Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
VisIt : A Tool for Visualizing and Analyzing Very Large Data Hank Childs, Lawrence Berkeley National Laboratory December 13, 2010.
2: Operating Systems Networking for Home & Small Business.
1 1.Log in to the computer in front of you –Temp account: 210class / 2.Update your in Cascadia's system –If I need to you I'll use.
| 1 EBSCOadmin EBSCO Support EDS Wiki Renata Wlodarczyk | EBSCO.
“This improved a lot since I started using Tango (three years ago) from scratch so I'm happy to see the efforts from the developers. Still there is room.
SciDAC SSS Quarterly Report Sandia Labs January 25, 2005 William McLendon Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed.
Version Control Systems
VisIt Project Overview
VisIt 2.0 Features Brad Whitlock.
VisIt Libsim Update DOE Computer Graphics Forum 2012 Brad Whitlock
In-situ Visualization using VisIt
Verification and Validation Unit Testing
Do anything from anywhere – tools to free your choice of OS
Azure Container Service
PyWBEM Python WBEM Client: Overview #2
Presentation transcript:

Large Scale Data Visualization with VisIt Hank Childs, Lawrence Berkeley, August 8, 2012 Basic usage Data analysis Derived quantities Scripting Moviemaking Comparisons File format readers In situ processing + more!

Tutorial Schedule  15 minutes: VisIt project overview (powerpoint)  45 minutes: Demonstration 20 minutesBasic features 5 minutesQueries 5 minutesScripting 5 minutesComparison 5 minutesMoviemaking 5 minutesAdvanced features  15 minutes:“How to succeed when you get home”  Questions  One-on-one time with presenters

VisIt is an open source, richly featured, turn-key application for large data.  Used by:  Visualization experts  Simulation code developers  Simulation code consumers  Popular  R&D 100 award in 2005  Used on many of the Top500  >>>100K downloads 217 pin reactor cooling simulation Run on ¼ of Argonne BG/P Image credit: Paul Fischer, ANL 1 billion grid points

Terribly Named!!! … intended for much more than just visualization Data Exploration Presentations Visual Debugging Analysis

 General techniques (e.g. integration, volumes, surface areas, etc.)  Specialized analysis (e.g. hohlraum flux at AGEX) Detector at AGEX Detector provided by VisIt (synthetic diagnostic) What sort of analysis is appropriate for VisIt?

VisIt has a rich feature set.  Meshes: rectilinear, curvilinear, unstructured, point, AMR  Data: scalar, vector, tensor, material, species  Dimension: 1D, 2D, 3D, time varying  Rendering (~15): pseudocolor, volume rendering, hedgehogs, glyphs, mesh lines, etc…  Data manipulation (~40): slicing, contouring, clipping, thresholding, restrict to box, reflect, project, revolve, …  File formats (~110)  Derived quantities: >100 interoperable building blocks  +,-,*,/, gradient, mesh quality, if-then-else, and, or, not  Many general features: position lights, make movie, etc  Queries (~50): ways to pull out quantitative information, debugging, comparative analysis

VisIt employs a parallelized client- server architecture.  Client-server observations:  Good for remote visualization  Leverages available resources  Scales well  No need to move data  Additional design considerations:  Plugins  Heavy use of VTK  Multiple UIs: GUI (Qt), CLI (Python), more… remote machine Parallel vis resources User data localhost – Linux, Windows, Mac Graphics Hardware You don’t have to run VisIt this way! You can run all on localhost (like this tutorial!) You can tunnel through ssh and run all on the remote machine

VisIt recently demonstrated good performance at unprecedented scale. ● Weak scaling study: ~62.5M cells/core 8 #coresProblem Size ModelMachine 8K0.5TIBM P5Purple 16K1TSunRanger 16K1TX86_64Juno 32K2TCray XT5JaguarPF 64K4TBG/PDawn 16K, 32K1T, 2TCray XT4Franklin Two trillion cell data set, rendered in VisIt by David Pugmire on ORNL Jaguar machine

It has taken a lot of research to make VisIt work Systems research: Adaptively applying algorithms in a production env. Algorithms research: How to efficiently calculate particle paths in parallel. Algorithms research: How to volume render efficiently in parallel. Methods research: How to incorporate statistics into visualization. Scaling research: Scaling to 10Ks of cores and trillions of cells. Architectural research: Hybrid parallelism + particle advection Systems research: Using smart DB technology to accelerate processing Architectural research: Parallel GPU volume rendering Algorithms research: Reconstructing material interfaces for visualization Algorithms research: Accelerating field evaluation of huge unstructured grids Don’t be scared! … VisIt developers mostly wear their software engineering hats – not their research hats.

The VisIt team focuses on making a robust, usable product for end users. Manuals – 300 page user manual – 200 page command line interface manual – “Getting your data into VisIt” manual Wiki for users (and developers) Revision control, nightly regression testing, etc Executables for all major platforms Day long class, complete with exercises Slides from the VisIt class

VisIt is a vibrant project with many participants.  Over 75 person-years of effort  Over 1.5 million lines of code  Partnership between: Department of Energy’s Office of Science, National Nuclear Security Agency, and Office of Nuclear Energy, the National Science Foundation XD centers (Longhorn XD and RDAV), and more… User community grows, including AWE & ASC Alliance schools Fall ‘06 VACET is funded Spring ‘08 AWE enters repo 2003 LLNL user community transitioned to VisIt R&D SciDAC Outreach Center enables Public SW repo 2007 Saudi Aramco funds LLNL to support VisIt Spring ‘07 GNEP funds LLNL to support GNEP codes at Argonne Summer‘07 Developers from LLNL, LBL, & ORNL Start dev in repo ‘07-’08 UC Davis & UUtah research done in VisIt repo 2000 Project started ‘07-’08 Partnership with CEA is developed 2008 Institutional support leverages effort from many labs More developers entering repo all the time

VisIt: What’s the Big Deal?  Everything works at scale  Robust, usable tool  Features that span the “power of visualization”:  Data exploration  Confirmation  Communication  Features for different kinds of users:  Vis experts  Code developers  Code consumers  Healthy future: vibrant developer and user communities

Before we begin…  Reminder: we will discuss file format issues, installation issues, and how to get help at the end of the tutorial  Important: ask questions any time!  Script located at visitusers.org under “short tutorial”…

 VisIt basics  Queries and expressions  Scripting  Moviemaking

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

Can I use a pre-built VisIt binary or do I need to build it myself?  Pre-built binaries work on most modern machines.  … but pre-built binaries are serial only.  Why the VisIt team can’t offer parallel binaries: Your MPI libraries, networking libraries are unlikely to match ours  … and it is difficult to use your own custom plugins with the pre-builts.  Recommendation: try to use the pre-builts first and build VisIt yourself if they don’t work.  Also: all VisIt clients run serial-only. If you want to install VisIt on your desktop to connect to a remote parallel machine, serial is OK.

How do I use pre-built VisIt binaries?  A: Go to

How do I use pre-built VisIt binaries?

Important

How do I use pre-built VisIt binaries?

How do I use the pre-built VisIt binaries?  Unix:  Download binary  Download install script  Run install script  --or—  Download binary  Untar  Mac:  Standard DMG install  Windows:  Download installer program & run  Full install notes:  Good for host profiles, maintaining multiple versions, multiple OSs Quick & easy

Important step: choosing host profiles  Many supercomputing sites have set up “host profiles”.  These files contain all the information about how to connect to their supercomputers and how to launch parallel jobs there.  You select which profiles to install when you install VisIt.  Profiles that come with VisIt:  NERSC, LLNL Open, LLNL Closed, ORNL, Argonne, TACC, LBNL desktop network, Princeton, UMich CAC  Other sites maintain profiles outside of VisIt repository.  If you know folks running VisIt in parallel at a site not listed above, ask them for their profiles.

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

Building VisIt from scratch  Building VisIt from scratch on your own is very difficult.  … but the “build_visit” script is fairly reliable.

What “build_visit” does  Downloads third party libraries  Patches them to accommodate OS quirks  Builds the third party libraries.  Creates “config-site” file, which communicates information about where 3 rd party libraries live to VisIt’s build system.  Downloads VisIt source code  Builds VisIt

“build_visit” details  There are two ways to use build_visit:  Curses-style GUI  Command line options through --console Developers all use --console and it shows!!  Tip:  Don’t build every third party library unless you really need to. Set up a “--thirdparty-path”.

“build_visit” details  Q1: How long does build_visit take? A: hours  Q2: I have my own Qt/VTK/Python, can I use those?  Hank highly recommends against  Q3: What happens after build_visit finishes?  A1: you can run directly in the build location  A2: you can make a package and do an install like you would with the pre-built binaries

“build_visit” details  Most common build_visit failures:  gcc is not installed  X11 development package is not installed  OpenGL development package is not installed  We should probably improve detection of this case, but we’re leery about false positives.  Most common VisIt runtime failure: really antique OpenGL drivers.  Hank runs SUSE 9.1 (from 2005) at home.  Build process for Windows is very different. Rarely a need to build on Windows, aside from VisIt development.

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

How to get your data into VisIt  There is an extensive (and up-to-date!) manual on this topic: “Getting Your Data Into VisIt”  Three ways:  Use a known format  Write a file format reader  In situ processing  Latter two covered in afternoon course

File formats that VisIt supports  ADIOS, BOV, Boxlib, CCM, CGNS, Chombo, CLAW, EnSight, ENZO, Exodus, FLASH, Fluent, GDAL, Gadget, Images (TIFF, PNG, etc), ITAPS/MOAB, LAMMPS, NASTRAN, NETCDF, Nek5000, OpenFOAM, PLOT3D, PlainText, Pixie, Shapefile, Silo, Tecplot, VTK, Xdmf, Vs, and many more  113 total readers  ile_formats_VisIt_supports  Some readers are more robust than others.  For some formats, support is limited to flavors of a file a VisIt developer has encountered previously (e.g. Tecplot).

File formats that VisIt supports  ADIOS, BOV, Boxlib, CCM, CGNS, Chombo, CLAW, EnSight, ENZO, Exodus, FLASH, Fluent, GDAL, Gadget, Images (TIFF, PNG, etc), ITAPS/MOAB, LAMMPS, NASTRAN, NETCDF, Nek5000, OpenFOAM, PLOT3D, PlainText, Pixie, Shapefile, Silo, Tecplot, VTK, Xdmf, Vs, and many more  Two readers that work common types of existing data:  BOV: raw binary data for rectilinear grid   you have a brick of data and you add an ASCII header that describes dimensions  PlainText: reads space delimited columns.  Controls for specifying column purposes

File formats that VisIt supports  ADIOS, BOV, Boxlib, CCM, CGNS, Chombo, CLAW, EnSight, ENZO, Exodus, FLASH, Fluent, GDAL, Gadget, Images (TIFF, PNG, etc), ITAPS/MOAB, LAMMPS, NASTRAN, NETCDF, Nek5000, OpenFOAM, PLOT3D, PlainText, Pixie, Shapefile, Silo, Tecplot, VTK, Xdmf, Vs, and many more  Common array writing libraries:  NETCDF: VisIt reader understands many (but not all) conventions  HDF5  Pixie is most general HDF5 reader  Many other HDF5 readers

File formats that VisIt supports  ADIOS, BOV, Boxlib, CCM, CGNS, Chombo, CLAW, EnSight, ENZO, Exodus, FLASH, Fluent, GDAL, Gadget, Images (TIFF, PNG, etc), ITAPS/MOAB, LAMMPS, NASTRAN, NETCDF, Nek5000, OpenFOAM, PLOT3D, PlainText, Pixie, Shapefile, Silo, Tecplot, VTK, Xdmf, Vs, and many more  Xdmf: specify an XML file that describes semantics of arrays in HDF5 file  VizSchema (Vs): add attributes to your HDF5 file that describes semantics of the arrays.

File formats that VisIt supports  ADIOS, BOV, Boxlib, CCM, CGNS, Chombo, CLAW, EnSight, ENZO, Exodus, FLASH, Fluent, GDAL, Gadget, Images (TIFF, PNG, etc), ITAPS/MOAB, LAMMPS, NASTRAN, NETCDF, Nek5000, OpenFOAM, PLOT3D, PlainText, Pixie, Shapefile, Silo, Tecplot, VTK, Xdmf, Vs, and many more  VTK: not built for performance, but it is great for getting into VisIt quickly  Silo: higher barriers to entry, but performs well and fairly mature

VTK File Format  The VTK file format has both ASCII and binary variants.  Great documentation at  Easiest way to write VTK files: use VTK modules  … but this creates a dependence on the VTK library  You can also try to write them yourself, but this is an error prone process.  Third option: visit_writer

VisItWriter writes VTK files  It is a “library” (actually a single C file) that writes VTK- compliant files.  The typical path is to link visit_writer into your code and write VTK files  There is also Python binding for visit_writer.  The typical path is to write a Python program that converts from your format to VTK  Both options are short term: they allow you to play with VisIt on your data. If you like VisIt, then you typically formulate a long term file format strategy.  More information on visit_writer: 

Python VisItWriter in action

Silo file format  Silo is a mature, self-describing file format that deals with multi-block data.  It has drivers on top of HDF5, NetCDF, and “PDB”.  Fairly rich data model  More information: 

Silo features

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

How to get help when you run into trouble  Six options:  FAQ  Documentation  VisIt-users mailing list  VisIt-users archives  VisIt users forum  VisIt-help-XYZ mailing list

FAQ:

Manuals & other documentation  Getting started manual  Users manual (old, but still useful)  Python interface  Getting Data Into VisIt  VisIt Class Slides  VisIt Class Exercises  This Tutorial

Visitusers.org  Users section has lots of practical tips:  “I solved this problem with this technique”  “Here’s my script to do this functionality”  In practical terms, this is a staging area for formal documentation in the future.

VisIt-users mailing list  You may only post to mailing list if you are also a subscriber  Approximately 400 recipients, approx. 300 posts per month.  Developers monitor mailing list, strive for 100% response rate  Response time is typically excellent (O(1 hour))  International community really participates … not unusual for a question from Australia to be answered by a European all while I’m asleep  List:  More information:  Archive:

VisIt User Forum  Increasingly popular option; you can post without receiving 300 s a month  But it is viewed by less people and less well supported.   Google searches these pages.

Visit-help-xyz  Some customer groups pay for VisIt funding and get direct support.  These customers can post directly to visit-help-xyz without being a subscriber  The messages are received by all VisIt developers and supported collectively  Lists:  Visit-help-asc, visit-help-scidac, visit-help-gnep, visit- help-ascem

How to get help when you run into trouble  Six options:  FAQ  Documentation  VisIt-users mailing list  VisIt-users archives  VisIt users forum  VisIt-help-XYZ mailing list

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

It is possible (although non-trivial) to write a custom user interface to VisIt

VisIt-PySide interfaces  PySide is a LGPL Python library that wraps Qt.  PySide is now distributed with VisIt and available from VisIt’s CLI.  Dynamically create and control Qt GUIs via Python.  Supports GUIs created with Qt’s Designer.

“How to make VisIt work after you get home”  How to get VisIt running on your machine  Downloading and installing VisIt  Building VisIt from scratch  How to get VisIt to read your data  How to get help when you run into trouble  I like the power of VisIt, but I hate the interface  How to run client-server

How to run client-server  There are two critical pieces:  Connecting to the remote machine  Getting an engine launched on the remote machine  This job is made substantially easier by host profiles.  (Demonstration)

Thank you for coming!!  Hank Childs,  Amit Chourasia,  User resources:  Main website:  Wiki:   Development resources:   SVN: