1. June 2010 2.0

2. Supervisors & Staff Supervisor: Oved Itzhak Developers: Saeed Mhameed Hani Ayoub

3. Reminder Dysis: Generic distributed system, to execute tasks remotely With Focus on: Transparency Generality Simplicity

4. Project Goal Improve the system! Make the system more General Performance improvements Improve management Reliability Consider Backward compatibility

5. Improvements Multitask support Improved Scheduler Executers queues monitoring and management Full fault tolerance

6. Multitask support Currently Support only one type of tasks simultaneously Disadvantage Not exploiting execution power Limiting the developer Less generic Goal Give the user the ability to execute any task without changing the current system state Example: Task1: Crawl WebPages Task2: Download WebPages Task3: process the result

7. Multitask support (cont.) Base Executer 1 Client Crawl Base Executer 1 Download Process Task 1 Crawl Task 2 Download Result1 Process Final Result Result2 Current System

8. Multitask support (cont.) Base Executer 1 Client Crawl Base Executer 1 Download Process Task 1 Crawl Task 2 Download Result1 Process Final Result Result2 Future System

9. Improved Scheduler Currently Simple RR Consider buffer load Goal Considers Buffer Size Executer machine power How? New scheduling algorithm based on:

10. Improved Scheduler (Cont.) Calculating executer power using # of processors Processor strength Benchmark Calculate Effective buffer size using Buffer size Executer power Choose best executer to have minimum effective buffer utilization

11. Monitoring and management Currently: No monitoring and management provided of tasks queue. Disadvantage: User cannot modify or monitor tasks in real-time Goal: Give the user the ability to: Add Remove Manipulate Change priority Monitor Reorder any waiting task.

12. Full Fault Tolerance Currently: Partial fault tolerance. i.e. the system reliable from executer failures and network disruption Goal: Make the system fully fault tolerance, so it’ll fulfill: Each task submitted to the system will eventually be executed and result will be received by the corresponding client. How? If task is not pending or executer is dead for a predefined timeout -> reschedule task.

13. Performance and reliability measurement The main improvements of the system are: Performance Reliability How will we measure these improvements? For performance: Run predefined benchmark on both current and future system, then compare results. For reliability: Create unreliable environment, then make sure all tasks are executed.

14. Q&A 2.0

15. Backup – Improved Scheduler - HOW? Definitions: Buffer Size (BS): current number of tasks Buffer Capacity (BC): number of tasks the buffer can hold Buffer Efficiency (BE): actual efficiency of tasks buffer BE = (BS \ BC) Executer Power (EP): Remote-machine strength Determined by: Processors number Processor frequency Benchmark time EP = (Processors_Number*(Processor_Frequency/1000) ) Banchmark_Time

16. Backup – Improved Scheduler - HOW? (Cont.) Definitions (cont.): Max Executer Power (Max_EP): Max_EP = Max { EPs } Power Percentage (PP): PP = (EP \ Max_EP) Effective Buffer Capacity (Eff_BC): Eff_BC = (BC * PP) Effective Buffer Efficiency (Eff_BE): Eff_BE = BS \ Eff_BC Scheduler(T) <= Max { PP( Min { Eff_BE } ) }