1. Increasing Performance of Commercial Reservoir Simulators by Core Skew Allocation
José S. A. Cavalcante Filho, Thomas D.S. Oliveira, Silvio R.R. Costa,
Luis V. M. Ribas, Margareth N. Cruz,
PETROBRAS S.A.
Daniel Dias, Ynigo Zamudio,
Schlumberger Ltd
Myrian C.A. Costa, Albino A. Aveleda, Alvaro L.G.A. Coutinho
High Performance Computing Center
COPPE-Federal University of Rio de Janeiro

2. What can we do if we can’t mess with the source code?
Advances in multi-core processors: how to increase performance?
Commercial software
Legacy software
No instrumentation nor code modification allowed

3. Issues on Multicore Performance
Multi-core regime: When the memory system is saturated, the order and pattern of data accesses becomes a performance determining factor.
"We varied both the number of nodes and the number of cores per node, and found that the efficiency (performance as a function of total cores) was largely independent of the number of nodes and extremely dependent on cores per node"
“Core skew” effects: Because each core might be in a slightly different phase of execution, different functions may be running on different cores at the same time
“Evaluation and Optimization of Multicore Performance Bottlenecks in Supercomputing Applications”, J. Diamond et al., 2011 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)

4. An Example: Rayleigh-Benard 4:1:1 - 501×125×125 mesh
Elements : 39,140,625
Nodes : 7,969,752
Edges : 43,833,636
Flow equations : 31,879,008
Temperature equations.: 7,642,824
Time steps : ,954
EdgeCFD solver on Marte, a Dell Cluster running 64 cores, MPI-P2P
Elias, R. N., Camata, J. J., Aveleda, A. A., Coutinho, A. L. G. A.,
Evaluation of Message Passing Communication Patterns in Finite Element Solution of Coupled Problems.
Lecture Notes in Computer Science, v p

5. Dell Cluster, 64 cores, core skew allocation
Communication graph
Similar results can be found in: Jeff Diamond, Byoung-Do Kim, Martin Burtscher, Steve Keckler, Keshav Pingali and Jim Browne, Multicore Optimization for Ranger,TeraGrid09,
Time spent in 10 time steps

6. Parallel Reservoir Simulator
ECLIPSE
The ECLIPSE* family of reservoir simulation software specialized in blackoil, compositional and thermal finite-volume reservoir simulation, as well as streamline reservoir simulation
More than 30 years in the market
ECLIPSE Black-oil simulation: three-phase, 3D reservoir simulation supporting extensive well controls, field operations planning, and comprehensive enhanced oil recovery (EOR) schemes.
"Schlumberger – Reservoir Simulation” -

7. ECLIPSE Features (cont’d)
ECLIPSE Compositional simulation: Describes reservoir fluid phase behavior and compositional changes associated with multi-component hydrocarbon flow. 
ECLIPSE FrontSim simulation: Models multiphase fluid flow along streamlines; enable better visualization of fluid flow in the reservoir.
ECLIPSE Thermal Simulation: Simulates a wide range of thermal recovery processes, including steam-assisted gravity drainage, toe to heel air injection, and cold heavy oil production with sand

8. ECLIPSE
"Schlumberger – Reservoir Simulation” -

9. New Generation of Parallel Reservoir Simulators
INTERSECT (IX)
Multistage parallel linear solver framework
Two-stage CPR1 (Constraint Pressure Residual) scheme for large-scale parallel runs
Parallel Algebraic Multigrid (PAMG) solver with a F-GMRES outer iteration as the first stage preconditioner and Parallel ILU-type scheme for the second stage
SPE “Parallel Scalable Unstructured CPR-Type Linear Solver for Reservoir Simulation”, H. Cao et al., SPE Annual Technical Conference and Exhibition, 9-12 October 2005, Dallas, Texas

10. INTERSECT (IX) Architectural Features
Static and dynamic load balance on unstructured and structured grids
Simulator architecture supports black-oil and compositional models within a general formulation
Distribution of data between available processors is determined by PARMETIS
Communication between processors is based on MPI (OOMPI)
"
SPE “An Extensible Architecture for Next Gaeneration Scalable Parallel Reservoir Simulation”, D. Debaun et al., 19th SPE Reservoir Simulation Symposium, 2005, The Woodlands, Texas

11. INTERSECT (IX) Architectural layers in the simulator system
SPE “An Extensible Architecture for Next Generation Scalable Parallel Reservoir Simulation”, D. Debaun et al., 19th SPE Reservoir Simulation Symposium, 2005, The Woodlands, Texas

12. Multicore Processors Nehalem Family Harpertown Family
3 way 1333MHz QPI memory access
8MB L3 cache shared by all cores
Smart cache LLC allocation
Allocation of 2 cores ????
Harpertown Family
1333MHz FSB memory access
12MB L2 cache shared by each pair of cores
Allocation of 2 cores???
Multicore bottlenecks:
L3 cache capacity
off-chip bandwidth
DRAM banks.

13. Benchmarks B1 – Benchmark 1 Black-oil model
Significant amounts of free gas and very thin cells
1 million active cells
B2 – Benchmark 2
Compositional model
Realistic reservoir
2 million cells

14. Benchmark B1
Horizontal permeability distribution in B1

15. Benchmark B1 - Output

16. Benchmark B2
Horizontal permeability distribution in B2

17. Benchmark B2 - Output

18. Clusters Marte Galileu DELL cluster PowerEdge M1000e
16 nodes: 128 cores
Intel Xeon E5450 (Harpertown)
256 GB RAM memory
InfiniBand 20Gbps DDR
Full clos topology
Galileu
SunBlade 6048
896 nodes: 7168 cores
Intel Xeon X5560 (Nehalem)
21 TB RAM memory
InfiniBand 40Gbps (QDR)
Torus-3D topology

19. Benchmark B1 - ECLIPSE
8 cores/node running in full-core mode needs 8 nodes
4 cores/node running in half-core mode needs 16 nodes

20. Benchmark B2 - ECLIPSE

21. Benchmark B1 – INTERSECT (IX)

22. Benchmark B2 – INTERSECT (IX)

23. Conclusions Core skew allocation reduces execution times in all cases
Effects are less pronounced in Nehalem
Reduction of hours or days in complex models on production runs
However, under utilization of resources is an economic issue
Need to incorporate new multi-core optimization techniques in the code