1. A Framework for Particle Advection for Very Large Data
Hank Childs, LBNL/UCDavis
David Pugmire, ORNL
Christoph Garth, Kaiserslautern
David Camp, LBNL/UCDavis
Sean Ahern, ORNL
Gunther Weber, LBNL
Allen Sanderson, Univ. of Utah

2. Particle advection is a foundational visualization algorithm
Advecting particles creates integral curves

3. Particle advection is the duct tape of the visualization world
Advecting particles is essential to understanding flow and other phenomena (e.g. magnetic fields)!

4. Outline A general system for particle-advection based analysis
Efficient advection of particles

5. Outline A general system for particle-advection based analysis
Efficient advection of particles

6. Goal
Efficient code for a variety of particle advection workloads and techniques
Cognizant of use cases from 1 particle to >>10K particles.
Need handling of every particle, every evaluation to be efficient.
Want to support many diverse flow techniques: flexibility/extensibility is key.
Fit within data flow network design (i.e. a filter)

7. Motivating examples of system
FTLE
Stream surfaces
Streamline
Dynamical Systems (e.g. Poincaré Maps)
Statistics based analysis
+ more

8. Design PICS filter: parallel integral curve system Execution:
Instantiate particles at seed locations
Step particles to form integral curves
Analysis performed at each step
Termination criteria evaluated for each step
When all integral curves have completed, create final output

9. Design Five major types of extensibility: How to parallelize?
How do you evaluate velocity field?
How do you advect particles?
Initial particle locations?
How do you analyze the particle paths?

10. Inheritance hierarchy
avtPICSFilter
avtIntegralCurve
Streamline Filter
Your derived type of PICS filter
avtStreamlineIC
Your derived type of integral curve
We disliked the “matching inheritance” scheme, but this achieved all of our design goals cleanly.

11. #1: How to parallelize? avtICAlgorithm avtParDomIC-Algorithm
(parallel over data)
avtSerialIC-Algorithm
(parallel over seeds)
avtMasterSlave-ICAlgorithm

12. #2: Evaluating velocity field
avtIVPField
avtIVPVTKField
avtIVPVTK- TimeVarying- Field
avtIVPM3DC1 Field
avtIVP-<YOUR>HigherOrder-Field
IVP = initial value problem

13. #3: How do you advect particles?
avtIVPSolver
avtIVPDopri5
avtIVPEuler
avtIVPLeapfrog
avtIVP-M3DC1Integrator
avtIVPAdams-Bashforth
IVP = initial value problem

14. #4: Initial particle locations
avtPICSFilter::GetInitialLocations() = 0;

15. #5: How do you analyze particle path?
avtIntegralCurve::AnalyzeStep() = 0;
All AnalyzeStep will evaluate termination criteria
avtPICSFilter::CreateIntegralCurveOutput( std::vector<avtIntegralCurve*> &) = 0;
Examples:
Streamline: store location and scalars for current step in data members
Poincare: store location for current step in data members
FTLE: only store location of final step, no-op for preceding steps
NOTE: these derived types create very different types of outputs.

16. Putting it all together
PICS Filter
::CreateInitialLocations() = 0;
avtICAlgorithm
avtIVPSolver
avtIVPField
Vector<
avtIntegral-Curve>
::AnalyzeStep() = 0;
Integral curves sent to other processors with some derived types of avtICAlgorithm.

17. Outline A general system for particle-advection based analysis
Efficient advection of particles

18. Advecting particles
Decomposition of large data set into blocks on filesystem ?
What is the right strategy for getting particle and data together?

19. Strategy: load blocks necessary for advection
Go to filesystem and read block
Decomposition of large data set into blocks on filesystem

20. Strategy: load blocks necessary for advection
Decomposition of large data set into blocks on filesystem

21. “Parallelize over Particles”
Basic idea: particles are partitioned over PEs, blocks of data are loaded as needed.
Positives:
Indifferent to data size
Trivial parallelization (partition particles over processors)
Negative:
Redundant I/O (both across PEs and within a PE) is a significant problem.

22. “Parallelize over data” strategy: parallelize over blocks and pass particles
PE1
PE2
PE3
PE4

23. “Parallelize over Data”
Basic idea: data is partitioned over PEs, particles are communicated as needed.
Positives:
Only load data one time
Negative:
Starvation!

24. Contracts are needed to enable these different processing techniques
Parallelize-over-seeds
One execution per block
Only limited data available at one time
Parallelize-over-data
One execution total
Entire data set available at one time

25. Both parallelization schemes have serious flaws.
Two approaches:
Parallelizing Over
I/O
Efficiency
Data
Good
Bad
Particles
Parallelize
over particles
over data
Hybrid algorithms

26. The master-slave algorithm is an example of a hybrid technique.
Uses “master-slave” model of communication
One process has unidirectional control over other processes
Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over-particles.
Nice property for production visualization tools.
(Implemented in VisIt)
D. Pugmire, H. Childs, C. Garth, S. Ahern, G. Weber, “Scalable Computation of Streamlines on Very Large Datasets.” SC09, Portland, OR, November, 2009

27. Master-Slave Hybrid Algorithm
Divide PEs into groups of N
Uniformly distribute seed points to each group P0 P1 P2 P3 P4 P5 P6 P7 P8 P9
P10
P11
P12
P13
P14
P15
Slave
Master
Master:
Monitor workload
Make decisions to optimize resource utilization
Slaves:
Respond to commands from Master
Report status when work complete

28. Master Process Pseudocode{
while ( ! done )
if ( NewStatusFromAnySlave() )
commands = DetermineMostEfficientCommand()
for cmd in commands
SendCommandToSlaves( cmd ) }
What are the possible commands?

29. Commands that can be issued by master
Assign / Loaded Block
Assign / Unloaded Block
Handle OOB / Load
Handle OOB / Send
OOB = out of bounds
Master
Slave
Setup nomenclature
Slave is given a particle that is contained in a block that is already loaded

30. Commands that can be issued by master
Assign / Loaded Block
Assign / Unloaded Block
Handle OOB / Load
Handle OOB / Send
OOB = out of bounds
Master
Slave
Slave is given a particle and loads the block

31. Commands that can be issued by master
Assign / Loaded Block
Assign / Unloaded Block
Handle OOB / Load
Handle OOB / Send
OOB = out of bounds
Master
Slave
Load
Slave is instructed to load a block. The particle advection in that block can then proceed.

32. Commands that can be issued by master
Assign / Loaded Block
Assign / Unloaded Block
Handle OOB / Load
Handle OOB / Send
OOB = out of bounds
Master
Slave
Send to J
Slave J
Slave is instructed to send the particle to another slave that has loaded the block

33. Master Process Pseudocode{
while ( ! done )
if ( NewStatusFromAnySlave() )
commands = DetermineMostEfficientCommand()
for cmd in commands
SendCommandToSlaves( cmd ) }

34. Driving factors… You don’t want PEs to sit idle
(The problem with parallelize-over-data)
You don’t want PEs to spend all their time doing I/O
(The problem with parallelize-over-particles)
 you want to do “least amount” of I/O that will prevent “excessive” idleness.

35. Heuristics If no Slave has the block, then load that block!
If some Slave does have the block…
If that Slave is not busy, then have it process the particle.
If that Slave is busy, then wait “a while.”
If you’ve waited a “long time,” have another Slave load the block and process the particle.

36. Master-slave in action
Iteration
Action
S0 reads B0,
S3 reads B1 1
S1 passes points to S0,
S4 passes points to S3,
S2 reads B0 S0 S1 S2 S3 S4
0: Read
1: Pass
1: Read

37. Algorithm Test Cases Core collapse supernova simulation
Magnetic confinement fusion simulation
Hydraulic flow simulation

38. Workload distribution in parallelize-over-data
Starvation
3 Procs doing all the work!!

39. Workload distribution in parallelize-over-particles
Too much I/O
Each streamline is computed by a single processor
Lots of redundant I/O

40. Workload distribution in master-slave algorithm
Just right
Good mix of processors working on different parts of the problem

41. Workload distribution in supernova simulation
Parallelization by:
Data
Particles
Hybrid
Interesting way to look at workload distribution
Colored by PE doing integration

42. Astrophysics Test Case: Total time to compute 20,000 Streamlines
Uniform Seeding
Non-uniform Seeding
Seconds
Seconds
COMPARISON
Hybrid algorithm outperforms both traditional methods
256M cells
512 blocks
Number of procs
Number of procs
Particles
Data
Hybrid

43. Astrophysics Test Case: Number of blocks loaded
Uniform Seeding
Non-uniform Seeding
Blocks loaded
Blocks loaded
Seed: Lots of redundant I/O
Block: Very low
Hybrid can adapt to do the right amount of I/O
Number of procs
Number of procs
Particles
Data
Hybrid

44. Summary: Master-Slave Algorithm
First ever attempt at a hybrid algorithm for particle advection
Algorithm adapts during runtime to avoid pitfalls of parallelize-over-data and parallelize-over- particles.
Nice property for production visualization tools.
Implemented inside VisIt visualization and analysis package.

45. Final thoughts… Summary:
Particle advection is important for understanding flow and efficiently parallelizing this computation is difficult.
We have developed a freely available system for doing this analysis for large data.
Documentation:
(PICS)
(VisIt)
Future work:
UI extensions, including Python
Additional analysis techniques (FTLE & more)

46. Acknowledgements
Funding: This work was supported by the Director, Office of Science, Office and Advanced Scientific Computing Research, of the U.S. Department of Energy under Contract No. DE- AC02-05CH11231 through the Scientific Discovery through Advanced Computing (SciDAC) program's Visualization and Analytics Center for Enabling Technologies (VACET).
Program Manager: Lucy Nowell
Master-Slave Algorithm: Dave Pugmire (ORNL), Hank Childs (LBNL/UCD), Christoph Garth (Kaiserslautern), Sean Ahern (ORNL), and Gunther Weber (LBNL)
PICS framework: Hank Childs (LBNL/UCD), Dave Pugmire (ORNL), Christoph Garth (Kaiserslautern), David Camp (LBNL/UCD), Allen Sanderson (Univ of Utah)

47. A Framework for Particle Advection for Very Large Data
Thank you!!
A Framework for Particle Advection for Very Large Data
Hank Childs, LBNL/UCDavis
David Pugmire, ORNL
Christoph Garth, Kaiserslautern
David Camp, LBNL/UCDavis
Sean Ahern, ORNL
Gunther Weber, LBNL
Allen Sanderson, Univ. of Utah