1. Phoenix: A Substrate for Resilient Distributed Graph Analytics
Roshan Dathathri          Gurbinder Gill
Loc Hoang         Keshav Pingali

2. Phoenix
Substrate to recover from fail-stop faults in distributed graph applications
Tolerates arbitrary number of failed machines, including cascading failures
Classifies graph algorithms and uses class-specific recovery protocol

3. Phoenix
Substrate to recover from fail-stop faults in distributed graph applications
Tolerates arbitrary number of failed machines, including cascading failures
Classifies graph algorithms and uses class-specific recovery protocol
No overhead in the absence of faults, unlike checkpointing
24x faster than GraphX
Evaluated on 128 hosts using graphs      1TB
Outperforms checkpointing when up to 16 hosts fail
Say GraphX is fault-tolerant distributed graph processing system

4. State of a graph Graph State of the graph A C E G B D F H ∞ ∞ ∞ ∞ ∞ ∞∞
State of the graph A B C D E F G H ∞

5. Distributed execution model
Graph
Host h1
Host h2 A C E G A C E E G
CuSP [IPDPS’19] B D F H B D D F H
State of the graph A B C D E F G H A B C D E D E F G H
Galois [SoSP’13] ∞ ∞ ∞
compute
compute
state transition
communicate
Gluon [PLDI’18] ∞ 1
Say bfs operator ∞ 1 ∞ 1 1

6. How to recover from crashes or fail-stop faults?
Graph
Host h1
Host h2 A C E G A C E E G B D F H B D D F H
State of the graph A B C D E F G H A B C D E D E F G H
communicate ∞ 2 1 ∞ 2 1
Phoenix
preserve
re-initialize ∞ 2 1 ∞ 2 1 1 1 ∞ ∞
Fault detected during synchronization

7. States during algorithm execution and recovery
Globally Consistent States
Initial State
Checkpoint-Restart
Fault
Phoenix
Valid States
Final State
All States

8. Classification of graph algorithms
Globally Consistent States       
s.t. x x
Self-stabilizing algorithms
Locally-correcting algorithms
Valid States 
s.t. x 
All States
Say bfs is an example for locally-correcting
Globally-consistent algorithms
Globally-correcting algorithms

9. Classes: examples and recovery
Self-stabilizing algorithms
Locally-correcting algorithms
Collaborative filtering
Belief propagation
Pull-style pagerank
Pull-style graph coloring
Recovery:
Reinitialize lost nodes
Breadth first search
Connected components
Data-driven pagerank
Topology-driven k-core
Recovery:
Reinitialize lost nodes
Globally-consistent algorithms
Globally-correcting algorithms
Betweenness centrality
Recovery:
Restart from last checkpoint
Residual-based pagerank
Data-driven k-core
Latent Dirichlet allocation
Recovery: ?

10. Problem: find k-core of an undirected graph
k-core: maximal subgraph where every node has degree at least k A C E G E G B D F H F H
Graph
3-core of the graph
Say k-core is used in graph coloring (give that as intuition)

11. k-core algorithm (globally-correcting)
If node is alive (1) and its degree < k, mark dead (0) and decrement neighbor’s degree A B C D E F G H 1 2 3 4 5 A C E G 1 1 2 3 5 4 B D F H 1 1 2 4 3 1 1 2 3
Graph
Algorithm execution

12. Phoenix recovery for k-core algorithm
Valid state: degree of every node should be the number of alive (1) neighbors
Any node can be alive (1) A B C D E F G H A B C D E F G H 1 1 2 3 4 5 4 3
Phoenix A C E G
Fault 1 1 1 2 3 5 4 2 1 4 5 3 B D F H 1 1 2 4 3 1 1 2 3
Graph
Algorithm execution

13. Phoenix substrate for recovery
Phoenix invoked when fail-stop fault detected
Arguments to Phoenix: depends on algorithm class
Re-initialization function
Re-computation function (globally-correcting)
Phoenix recovery:
Re-initialize and synchronize proxies
Re-compute and synchronize proxies (optional)
Locally-correcting algorithm

14. Experimental setup Systems: Benchmarks: D-Galois Phoenix in D-Galois
Checkpoint-Restart (CR) in D-Galois
GraphX [GRADES’13]
Benchmarks: 
Connected components (cc)
K-core (kcore)
Pagerank (pr)
Single source shortest path (sssp) 
Inputs
twitter
rmat28
kron30
clueweb
wdc12
|V|
51M
268M
1073M
978M
3,563M
|E| 2B 4B
11B
42B
129B
|E|/|V| 38 16 44 36
Size (CSR)
16GB
35GB
136GB
325GB
986GB
Clusters
Stampede
Wrangler
No. of hosts
128 32
Machine
Intel Xeon Phi KNL
Intel Xeon Haswell
Each host
272 threads of KNL
48 threads of Haswell
Memory
96GB DDR3
128GB DDR4
Say algorithm class for each benchmark and why that algorithm was chosen

15. Wrangler: fault-free total time on 32 hosts
Speedup (log scale)
Geometric mean:
24x

16. Stampede: fault-free execution time on 128 hosts
Execution Time (s)
D-Galois and Phoenix are identical
Geometric mean overheads: CR-50: 31%                   CR-500: 8%

17. Stampede: execution time when faults occur on 128 hosts
pr on wdc12
Speedup of Phoenix over CR-50
Speedup of Phoenix over CR-500
Say Phoenix can be used with checkpoint 

18. Stampede: execution time overhead when faults occur
Recovery time of Phoenix is negligible
Compared to fault-free execution of Phoenix, when faults occur on 128 hosts:
System
Number of crashed machines
Average execution time overhead
Phoenix 4
14% 16
21% 64
44%
CR-50
Any
49%
CR-500
59%

19. Fail-stop fault-tolerant distributed graph systems
Globally-correcting algorithms?
Globally-consistent algorithms?
No fault-free execution overhead?
Tolerates any number of failed machines?
Guarantees precise results?
No programmer input?
GraphX [GRADES’13]  x
Imitator [DSN’14]
Zorro [SoCC’15]
CoRAL [ASPLOS’17]
Phoenix

20. Future Work
Extend Phoenix to handle data corruption errors or byzantine faults
Use compilers to generate Phoenix recovery functions automatically
Explore Phoenix-style recovery for other application domains

21. Conclusion
Phoenix: substrate to recover from fail-stop faults in distributed graph applications
Recovery protocols based on classification of graph algorithms
Implemented in D-Galois, the state-of-the-art distributed graph system
Evaluated on 128 hosts using graphs TB
No overhead in the absence of faults, unlike checkpointing
Outperforms checkpointing when up to 16 hosts crash

22. Programmer effort for Phoenix
Globally-correcting kcore and pr:
1 day of programming
150 lines of code added (to 300 lines of code)
Locally-correcting cc and sssp:
Negligible programming effort
30 lines of code added

23. Phoenix substrate for recovery: globally-correcting

24. Stampede: execution time when faults occur on 128 hosts
cc on wdc12
Speedup of Phoenix over CR-50
Speedup of Phoenix over CR-500

25. Stampede: execution time when faults occur on 128 hosts
kcore on wdc12
Speedup of Phoenix over CR-50
Speedup of Phoenix over CR-500

26. Stampede: execution time when faults occur on 128 hosts
sssp on wdc12
Speedup of Phoenix over CR-50
Speedup of Phoenix over CR-500