6/11/2018 Finding Oil with Cells: Seismic Imaging Using a Cluster of Cell Processors Michael Perrone IBM Master Inventor Mgr, Multicore Computing, IBM Research

Outline Problem description Algorithmic solution Cell implementation issues Conclusions

150M$ 60-70% Challenges “Easy” oil is gone Focus on imaging 6/11/2018 Challenges 150M$ 60-70% “Easy” oil is gone Focus on imaging Mathematics is known Computer power isn’t affordable yet

Salt Domes Gulf of Mexico Deep sea ~50B barrels Difficult to image! 6/11/2018 Salt Domes Gulf of Mexico Deep sea ~50B barrels Difficult to image! http://www-sst.unil.ch/research/seismic/G_Mexico.htm

6/11/2018 Sample Image

Snell’s Law V1 V2 V3 a a Reflected wave Incident wave Transmitted wave Boundaries

Seismic Data Collection 6/11/2018 Seismic Data Collection Air Gun Surface Receiver Buried reflectors

Seismic Data Collection 6/11/2018 Seismic Data Collection Air Gun Surface Receiver Buried reflectors

Seismic Data Collection 6/11/2018 Seismic Data Collection Air Gun Surface Receiver Buried reflectors Internal Reflections

Ocean Seismic Survey 4, 8, 16 or more streamers 6/11/2018 Ocean Seismic Survey 4, 8, 16 or more streamers Receiver every ~25 meters Dragged across survey area Streamers Ship 1 km 5 km 1 Shot

Ocean Seismic Survey Multiple passes Drift Noise Equipment failure 6/11/2018 Ocean Seismic Survey Multiple passes Drift Noise Equipment failure 10-20TB per survey >100K Shots recorded

Outline Problem description Algorithmic solution Cell implementation issues Conclusions

RTM - Reverse Time Migration (Biondi & Shan, 2002) 6/11/2018 RTM - Reverse Time Migration (Biondi & Shan, 2002) Use 3D wave equation to model sound in Earth [ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t) Forward Propagation: F(x,y,z,t) Reverse Propagation: R(x,y,z,t) Image generation I(x,y,z) = ∑t F(x,y,z,t) R(x,y,z,t)

Outline Problem description Algorithmic solution Cell implementation issues Conclusions

Implementing the Wave Equation 6/11/2018 Implementing the Wave Equation Finite difference in time: µ∂2t P(x,y,z,t) ≈ P(x,y,z,t+1) - 2P(x,y,z,t) + P(x,y,z, t-1) Finite difference in space: ∂2x P(x,y,z,t) ≈ ∑ngx(n)P(x+n,y,z,t) ∂2y P(x,y,z,t) ≈ ∑ngy(n)P(x,y+n,z,t) ∂2z P(x,y,z,t) ≈ ∑ngz(n)P(x,y,z+n,t) Absorbing boundary conditions

Data Flow Hide disk access behind computes Re-use data - Tiling 6/11/2018 Data Flow Hide disk access behind computes Re-use data - Tiling Multibuffer DMAs Load Balancing Main memory (each iteration) Load V(x,y,z) P(x,y,z,t) P(x,y,z,t-1) Store P(x,y,z,t+1) Disk (every Nth iterations) Store P(x,y,z,t) - or - Load P(x,y,z,t)

RTM Algorithm (for each shot) 6/11/2018 Image RTM Algorithm (for each shot) Load data Velocity model v(x,y,z) Source & Receiver data Forward propagation Calculate F(x,y,z,t) Every N timesteps Compress F(x,y,x,t) Write F(x,y,x,t) to disk/memory Backward propagation Calculate R(x,y,z,t) Read F(x,y,x,t) from disk/memory Decompress F(x,y,x,t) Calculate partial sum of I(x,y,z) Merge I(x,y,z) with global image t=N F(N) R(N) I(N) t=2N F(2N) R(2N) I(2N) t=3N F(3N) R(3N) I(3N) t=kN F(kN) R(kN) I(kN) . . . . . . . . .

Data Partitioning Hides Latency Simple split on XZ-plane Boundary communication with DMAs or MPI Send boundary Compute bulk Receive boundary Compute boundary Data Partitioning Hides Latency X 1 2 Y Z

Load Balancing: Source Data “Spray” Full Velocity Model X Shot Y Spray Velocity Subset Problem: BW bottleneck “Spray” shot data onto grid Irregular data access Z direction stride 1 Solution: Improve data access Sort data points Calculate once per shot Adjust partitioning to balance

Data Compression Compression Quality Lossy Max/Min scaling “Global” “Local” Z Compression Lossy Max/Min scaling 4x Reduced BW requirement Quality Signal decays with depth Local version gives better signal resolution

RTM Parameters Survey size Velocity model Migration iterations 6/11/2018 RTM Parameters Survey size Frequency: ~5 shots / minute Shots: >100K per survey Survey size: 10-20TB Wall time: ~1 month Velocity model Multiple square miles: 2x2, 5x5, 10x10, etc. 1-10GB Migration iterations Thousands Depends on depth required

“Master/Slave” Cell Cluster Building Blocks 6/11/2018 “Master/Slave” Cell Cluster Building Blocks SRW248 Compute racks (72 QS22’s) plus storage rack

Outline Problem description Algorithmic solution Cell implementation issues Conclusions

[ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t) Conclusions 150M$ 60-70% Built 297-node Cell cluster Demonstrated Effectiveness of Master/Slave model Data flow issues can be managed Significant speed-up over Intel (4-10x depending on data) Currently generating real seismic images! Will be used to determine drilling sites…. [ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t)

6/11/2018 BACKUP SLIDES

6/11/2018 Abstract Modern deep sea oil exploration is a very expensive proposition. A single well can cost about 150M$ and the probability of drilling a "dry" hole is about 60-70 percent! To help reduce cost, oil companies have turned to increasingly complex computational imaging techniques to improve the quality of imaging. And in order to reduce the "time to oil", images must be generated as quickly as possible; so these algorithms are run on high-performance computing clusters. This presentation will discuss one such imaging application implemented on a 296-node, heterogeneous cluster composed primarily of Cell processors.