MapReduce-Simulator: Matchmaking and Scheduling Algorithm SYSC 5807: Resource Management on Distributed Systems MapReduce-Simulator: Matchmaking and Scheduling Algorithm Norman Lim Dept. of Systems and Computer Engineering Carleton University Ottawa, ON, Canada nlim@sce.carleton.ca

Overview of Matchmaking and Scheduling (Mapping) Algorithm 1. When a job j arrives, it is placed in a queue depending on the scheduling policy selected. FIFO: uses first-in first-out queue EDF: uses earliest deadline first queue 2. Job/Task Mapping algorithm is invoked. Retrieves the first job in the queue and maps each of the job’s tasks onto the resources. Maps each task t onto a resource r that can execute t at its earliest possible time (at or after its earliest start time). For a particular job, the tasks in that job with higher execution time are mapped first.

Overview of Matchmaking and Scheduling (Mapping) Algorithm Cont. EDF Policy Only 3. If a job j is not able to be scheduled to complete executing before its deadline, then the Job/Task Re-mapping algorithm is invoked. Remaps j and the set of jobs (called S) that have caused j to miss its deadline. S includes jobs that are scheduled to start at, or complete within the interval: [start time of job j, deadline of job j]. j and the jobs in S are re-ordered according to earliest deadline first. The Job/Task Mapping algorithm (see previous slide) is invoked to map j and the jobs in S.

Example: Jobs and Resources Job 1 (j1) Arrival time = 0s EST = 2s Deadline = 30s t1 2 t2 5 t3 8 t4 10 t5 11 Job 2 (j2) Arrival time = 1s EST = 1s Deadline = 20s t1 4 t2 5 t3 10 Resource 1 (r1) Map Capacity = 1 Reduce Capacity = 1 Resource 2 (r2) Map Capacity = 1 Reduce Capacity = 1 Two jobs: j1 and j2. DAG shows the jobs’ tasks. - Blue circles: Map tasks - Red circles: Reduce tasks The task id and its required execution time are shown in the Two resources: r1 and r2. EST = earliest start time

Example: FIFO Job 1 (j1) Arrival time = 0s EST = 2s Deadline = 30s FIFO Queue map j1, t2 j1, t1 j2, t2 j1, t4 j2, t3 reduce j2 will miss its deadline. Job 1 arrives Job 2 arrives Job 1 gets scheduled Job 2 gets scheduled. map j1, t3 j2, t1 j1, t5 reduce

Example: EDF (Case 1) Job 1 (j1) Arrival time = 0s EST = 2s Deadline = 30s Job 2 (j2) Arrival time = 1s EST = 1s Deadline = 20s EDF Queue map j2, t1 j1, t3 j1, t5 reduce Job 1 arrives. Job 2 arrives, and goes to the start of the queue. Job 1 and 2 are scheduled. Note: In this case, Job 1 is not scheduled immediately because the scheduler was busy. map j2, t2 j1, t2 j1, t1 j2, t3 j1, t4 reduce

Example: EDF (Case 2) Job 2 (j2) Arrival time = 1s EST = 1s Deadline = 20s Job 1 (j1) Arrival time = 0s EST = 2s Deadline = 30s Job 1 (j1) Arrival time = 0s EST = 2s Deadline = 30s Job 2 (j2) Arrival time = 1s EST = 1s Deadline = 20s EDF Queue map j2, t1 j1, t2 j1, t3 j1, t1 j2, t2 j1, t5 j1, t4 j2, t3 reduce j2 will miss its deadline. Job 1 arrives, and gets scheduled. Note: In this case, Job 1 is scheduled immediately because the scheduler was not busy. Job 2 arrives, and gets scheduled. Job 2 is not able to finish its execution before its deadline so the Job/Task Re-mapping algorithm is executed to remap the jobs j1 and j2. This time, Job 2 (earlier deadline) is scheduled first, followed by Job 1. map j2, t2 j1, t3 j1, t2 j2, t1 j1, t1 j2, t3 j1, t5 j1, t4 reduce