Cluster 2004 San Diego, CA A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison September 23 rd, 2004

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 2 Grid Trivia  How many of you have submitted a job to the Grid resources and did never hear back from it?  How many of you got mad by the inconsistent behavior of some grid resources? Completing successfully some jobs and failing others.. Similar jobs performing completely different..... We did!

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 3 Goal: Prevent Unexpected Behavior in a Grid  Learn from experience and prevent them from repeating in the future again.  Causes for unexpected behavior in a Grid: Black holes Resources with –Faulty hardware –Buggy or misconfigured software Extremely slow computational sites Memory leaks..etc

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 4 Black holes

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 5 Black holes  Definition: “A black hole is a region of spacetime from which nothing can escape, even light.”  If you send a light beam to a black hole, you never hear back from it.  You can only know it after you have encounter it. Is it too late? No. You should learn from experience..

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 6 Black holes in the Grid  Resources that accept jobs but never complete them You send a job to a resource, but never hear back from it.

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 7 Black hole examples from real life:  In the WCER educational video processing pipeline: A specific pool was accepting and processing our jobs for a couple of hours, but evicting before completion. A machine accepted a job, but due to a memory leak it kept throwing “shadow exceptions” and retrying the job forever. Some thirdparty (GridFTP, DiskRouter) transfers hang occasionally and never returned. A machine caused an error because of a corrupted FPU. It successfully completed MPEG-1 encoding but failed MPEG-4.

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 8 Grid is good.. but not perfect..  Heterogeneous resources  Multi administrative domains  Spanning wide area networks  Consists of commodity hardware and software Prone to network-, hardware-, software-, middleware- failures! We cannot expect everything from the Grid or Grid middleware!

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 9 Take the Ethernet Approach  A truly distributed (and very effective) access control protocol to a shared service  Client responsible access control  Client responsible for error detection  Client responsible for fairness Keep track of job/resource performance & failure characteristics as observed by the client. Use job/user log files collected at the client side to build a grid knowledgebase.

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 10 Grid Knowledgebase  Parse user/job log files  Load them into a database  Aggregate experience of different jobs  Interpret them  Plan action  Generate feedback to the scheduler as well as to the user

JOB DESCRIPTIONS PLANNER JOB QUEUE JOB LOGS GRID RESOURCES JOB SCHEDULER MATCH MAKER Personal Computers Storage Servers Clusters

JOB DESCRIPTIONS PLANNER JOB QUEUE JOB LOGS GRID RESOURCES JOB SCHEDULER MATCH MAKER Personal Computers Storage Servers Clusters JOB PARSER GRID KNOWLEDGEBASE DATABASEDATA MINER NOTIFICATION LAYER ADAPTATION LAYER

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 13 Database Schema FieldType JobIdInt JobNamestring StateInt SubmitHoststring SubmitTimeInt ExecuteHoststring [] ExecuteTimestring [] ImageSizeint[] ImageSizeTimeint [] EvictTimeint [] Checkpointedbool [] EvictReasonstring TerminateTimeint [] TotalLocalUsagestring TotalRemoteUsagestring TerminateMessagestring ExceptionTimeint [] ExceptionMessagestring [] Evicted Exception Submit Terminated Abnormally Terminated Normally Schedule Execute Exit code = 0? Job Succeeded Job Failed Yes No Suspend Un-suspend User

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 14 Difference from existing approaches  Client view  Use only job/user log files at the client side Many administrators do not want to share resource/scheduler log files.  We do not need to know everything going on in the whole grid Scalable

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 15 What do we get?  Collecting job execution time statistics Average job execution time Standard deviation Fit a distribution  Detect and avoid black holes For normal distribution: –99.7% of job execution times should lie between (avg-3*stdev) and (avg+3*stdev) –96% of job execution times should lie between (avg-2*stdev) and (avg+2*stdev)

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 16 Detecting hanging transfers

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 17 Setting Execution Time Limits  Avg = 7.8 min  Stdev = 3.17min  For normal distribution: %99.7 : [0 – min] %96 : [1.46 min – min]

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 18 What do we get? (2)  Identifying misconfigured machines e.g. find set of machines which fail jobs with I/O data size larger than 2 GB (i.e. OS limitations)  Identifying factors affecting job run-time  Bug hunting Job failures on certain inputs Memory leaks –Scheduler logs image size regularly

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 19 Catching Memory Leaks Job Memory Image Size (MB) Time

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 20 What do we get? (3)  Application optimization How long does each step of an application/pipeline take to execute?  Adaptation Find resources that take least time to execute jobs from a particular class

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 21 Conclusions  View of the Grid from the client side  Job/user log files as main source of information  Aggregate experience of different jobs and pass them to future ones  Helps in: Catching black holes Identify faulty/misconfigured resources Bug tracking Statistics collection  Future work: Merge experience of different clients

A Client-centric Grid Knowledgebase George Kola, Tevfik Kosar and Miron Livny 22 Thank you… For more information, contact: Tevfik Kosar