1. Lu Qin Center of Quantum Computation and Intelligent Systems, University of Technology, Australia Jeffery Xu Yu The Chinese University of Hong Kong, China Lijun Chang The University of New South Wales, Australia Hong Cheng The Chinese University of Hong Kong, China Xuemin Lin The University of New South Wales, Australia East China Normal University, China Mahmoud Agbareya, 13 January 2015

2. Agenda Introduction The Scalable Graph Processing Class ( SGC ) SGC Algorithms Performance Studies 2

3. Agenda Introduction The Scalable Graph Processing Class ( SGC ) SGC Algorithms Performance Studies 3

4. Introduction What is MapRecuce? MapReduce Class ( MRC ) Minimal MapReduce Class ( MMC ) 4

5. Introduction What is MapRecuce? MapReduce Class ( MRC ) Minimal MapReduce Class ( MMC ) 5

6. What is MapReduce Programming model for processing large data sets in distributed systems. Process the data as (key, value) pairs. May executes in rounds Each round has three phases: map, shuffle and reduce. Each round runs in many machines – each machine is dedicated for one task (map or reduce) Introduced by two researchers from Google in 2004. Most popular implementation is Hadoop. 6

7. What is MapReduce (cont.) Example 7

8. Introduction What is MapRecuce? MapReduce Class ( MRC ) Minimal MapReduce Class ( MMC ) 8

9. MapReduce Class ( MRC ) Definition 9

10. 10 MapReduce Class ( MRC ) (Graph version) Definition

11. Introduction What is MapRecuce? MapReduce Class ( MRC ) Minimal MapReduce Class ( MMC ) 11

12. Minimal MapReduce Class ( MMC ) Definition 12

13. 13 Minimal MapReduce Class ( MMC ) Definition

14. Agenda Introduction The Scalable Graph Processing Class ( SGC ) SGC Algorithms Performance Studies 14

15. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 15

16. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 16

17. Motivation 17

18. Motivation We aim to define a MapReduce class in which, graph algorithm has the following three properties: Scalability: The algorithm can always be speeded up by adding more machines. Stability: The algorithms stops in bounded number of rounds. Robustness: The algorithm never fails regardless of how much memory each machine has. 18

19. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 19

20. Preliminaries 20

21. Preliminaries 21

22. Preliminaries 22

23. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 23

24. 24 Scalable Graph Processing Class ( SGC ) definition

25. 25 Scalable Graph Processing Class ( SGC ) definition

26. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 26

27. Two graph operators in SGC 27

28. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 28

29. NE Join 29

30. NE Join 30

31. NE Join in MapReduce 31

32. NE Join in MapReduce 32

33. The Scalable Graph Processing Class ( SGC ) Motivation Preliminaries SGC Definition Two graph operators in SGC : NE Join EN Join 33

34. EN Join 34

35. EN Join 35

36. EN Join 36

37. EN Join in MapReduce 37

38. Agenda Introduction The Scalable Graph Processing Class ( SGC ) SGC Algorithms Performance Studies 38

39. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 39

40. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 40

41. Breadth First Search 41

42. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 42

43. Page Rank 43

44. Page Rank 44

45. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 45

46. Graph Keyword Search 46

47. Graph Keyword Search 47

48. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 48

49. 49 Forest Initializing Conditional Star Hooking Unconditional Star Hooking Pointer Jumping Star Detection Procedure

50. Connected Component 50 Forest Initializing: Line 1: find the minimum neighbor for each node and set it to be the parent.

51. Connected Component 51 Forest Initializing:

52. Connected Component 52

53. Connected Component 53 Forest Initializing:

54. Connected Component 54 Star Detection: Rules to detect that node is not in star (applied in order)

55. Connected Component 55

56. Connected Component 56

57. Connected Component 57

58. Connected Component 58

59. Connected Component 59 Conditional Star Hooking (inside the loop):

60. Connected Component 60 Conditional Star Hooking (inside the loop): After Conditional Star Hooking, it’s guaranteed that there are no edges between two starts.

61. Connected Component 61

62. Connected Component 62 Unconditional Star Hooking (inside the loop):

63. Connected Component 63

64. Connected Component 64 Pointer Jumping (inside the loop):

65. Connected Component 65

66. SGC Algorithms Basic Graph Algorithms: Breadth First Search Page Rank Graph Keyword Search Advanced Algorithms: Connected Component Minimum Spanning Forest 66

67. Minimum Spanning Forest 67

68. Minimum Spanning Forest 68

69. 69 Forest Initializing Cycle Breaking Edge Hooking Pointer Jumping

70. Minimum Spanning Forest 70

71. Minimum Spanning Forest Forest Initialization 71

72. Minimum Spanning Forest 72

73. Minimum Spanning Forest Cycle Breaking 73

74. Minimum Spanning Forest Pointer Jumping 74

75. Minimum Spanning Forest 75

76. Minimum Spanning Forest 76

77. Minimum Spanning Forest 77

78. Minimum Spanning Forest 78

79. Minimum Spanning Forest Edge Hooking 79

80. Minimum Spanning Forest 80

81. Agenda Introduction The Scalable Graph Processing Class ( SGC ) SGC Algorithms Performance Studies 81

82. 82

83. 83

84. 84

85. 85

86. 86

87. 87