1. 6/11/2018
Finding Oil with Cells: Seismic Imaging Using a Cluster of Cell Processors
Michael Perrone
IBM Master Inventor
Mgr, Multicore Computing, IBM Research

2. Outline
Problem description
Algorithmic solution
Cell implementation issues
Conclusions

3. 150M$ 60-70% Challenges “Easy” oil is gone Focus on imaging
6/11/2018
Challenges
150M$
60-70%
“Easy” oil is gone
Focus on imaging
Mathematics is known
Computer power isn’t affordable yet

4. Salt Domes Gulf of Mexico Deep sea ~50B barrels Difficult to image!
6/11/2018
Salt Domes
Gulf of Mexico
Deep sea
~50B barrels
Difficult to image!

5. 6/11/2018
Sample Image

6. Snell’s Law V1 V2 V3 a a Reflected wave Incident wave Transmitted wave
Boundaries

7. Seismic Data Collection
6/11/2018
Seismic Data Collection
Air Gun
Surface
Receiver
Buried reflectors

8. Seismic Data Collection
6/11/2018
Seismic Data Collection
Air Gun
Surface
Receiver
Buried reflectors

9. Seismic Data Collection
6/11/2018
Seismic Data Collection
Air Gun
Surface
Receiver
Buried reflectors
Internal Reflections

10. Ocean Seismic Survey 4, 8, 16 or more streamers
6/11/2018
Ocean Seismic Survey
4, 8, 16 or more streamers
Receiver every ~25 meters
Dragged across survey area
Streamers
Ship
1 km
5 km
1 Shot

11. Ocean Seismic Survey Multiple passes Drift Noise Equipment failure
6/11/2018
Ocean Seismic Survey
Multiple passes
Drift
Noise
Equipment failure
10-20TB per survey
>100K Shots recorded

12. Outline
Problem description
Algorithmic solution
Cell implementation issues
Conclusions

13. RTM - Reverse Time Migration (Biondi & Shan, 2002)
6/11/2018
RTM - Reverse Time Migration (Biondi & Shan, 2002)
Use 3D wave equation to model sound in Earth
[ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t)
Forward Propagation: F(x,y,z,t) Reverse Propagation: R(x,y,z,t)
Image generation
I(x,y,z) = ∑t F(x,y,z,t) R(x,y,z,t)

14. Outline
Problem description
Algorithmic solution
Cell implementation issues
Conclusions

15. Implementing the Wave Equation
6/11/2018
Implementing the Wave Equation
Finite difference in time: µ∂2t P(x,y,z,t) ≈ P(x,y,z,t+1) - 2P(x,y,z,t) + P(x,y,z, t-1)
Finite difference in space: ∂2x P(x,y,z,t) ≈ ∑ngx(n)P(x+n,y,z,t)
∂2y P(x,y,z,t) ≈ ∑ngy(n)P(x,y+n,z,t)
∂2z P(x,y,z,t) ≈ ∑ngz(n)P(x,y,z+n,t)
Absorbing boundary conditions

16. Data Flow Hide disk access behind computes Re-use data - Tiling
6/11/2018
Data Flow
Hide disk access behind computes
Re-use data - Tiling
Multibuffer DMAs
Load Balancing
Main memory (each iteration)
Load
V(x,y,z)
P(x,y,z,t)
P(x,y,z,t-1)
Store P(x,y,z,t+1)
Disk (every Nth iterations)
Store P(x,y,z,t) - or -
Load P(x,y,z,t)

17. RTM Algorithm (for each shot)
6/11/2018
Image
RTM Algorithm (for each shot)
Load data
Velocity model v(x,y,z)
Source & Receiver data
Forward propagation
Calculate F(x,y,z,t)
Every N timesteps
Compress F(x,y,x,t)
Write F(x,y,x,t) to disk/memory
Backward propagation
Calculate R(x,y,z,t)
Read F(x,y,x,t) from disk/memory
Decompress F(x,y,x,t)
Calculate partial sum of I(x,y,z)
Merge I(x,y,z) with global image
t=N F(N) R(N) I(N)
t=2N F(2N) R(2N) I(2N)
t=3N F(3N) R(3N) I(3N)
t=kN F(kN) R(kN) I(kN)
. . .
. . .
. . .

18. Data Partitioning Hides Latency Simple split on XZ-plane
Boundary communication with DMAs or MPI
Send boundary
Compute bulk
Receive boundary
Compute boundary
Data Partitioning
Hides Latency X 1 2 Y Z

19. Load Balancing: Source Data “Spray”
Full Velocity Model X
Shot Y
Spray
Velocity Subset
Problem: BW bottleneck
“Spray” shot data onto grid
Irregular data access
Z direction stride 1
Solution: Improve data access
Sort data points
Calculate once per shot
Adjust partitioning to balance

20. Data Compression Compression Quality Lossy Max/Min scaling
“Global”
“Local” Z
Compression
Lossy Max/Min scaling 4x
Reduced BW requirement
Quality
Signal decays with depth
Local version gives better signal resolution

21. RTM Parameters Survey size Velocity model Migration iterations
6/11/2018
RTM Parameters
Survey size
Frequency: ~5 shots / minute
Shots: >100K per survey
Survey size: 10-20TB
Wall time: ~1 month
Velocity model
Multiple square miles: 2x2, 5x5, 10x10, etc.
1-10GB
Migration iterations
Thousands
Depends on depth required

22. “Master/Slave” Cell Cluster Building Blocks
6/11/2018
“Master/Slave” Cell Cluster Building Blocks
SRW248
Compute racks (72 QS22’s) plus storage rack

23. Outline
Problem description
Algorithmic solution
Cell implementation issues
Conclusions

24. [ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t)
Conclusions
150M$
60-70%
Built 297-node Cell cluster
Demonstrated
Effectiveness of Master/Slave model
Data flow issues can be managed
Significant speed-up over Intel (4-10x depending on data)
Currently generating real seismic images!
Will be used to determine drilling sites….
[ ∂2t - v2(x,y,z) (∂2x + ∂2y + ∂2z) ] P(x,y,z,t) = S(x,y,z,t)

25. 6/11/2018
BACKUP SLIDES

26. 6/11/2018
Abstract
Modern deep sea oil exploration is a very expensive proposition. A single well can cost about 150M$ and the probability of drilling a "dry" hole is about percent! To help reduce cost, oil companies have turned to increasingly complex computational imaging techniques to improve the quality of imaging. And in order to reduce the "time to oil", images must be generated as quickly as possible; so these algorithms are run on high-performance computing clusters. This presentation will discuss one such imaging application implemented on a 296-node, heterogeneous cluster composed primarily of Cell processors.