SARA Reken- en Netwerkdiensten ToPoS: High-Throughput Parallel Processing Pipelines on the Grid Pieter van Beek SARA Computing and Networking Services High Performance Computing and Visualization e-Science Support ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten SARA Computing & Networking services SARA hosts the Dutch (NCF) supercomputer SARA/UvA have set up the E-bioLab CAVE used for 3D visualisation SARA provides infrastructure and support for academic projects and institutions. Tier-1 site for LHC at CERN

SARA Reken- en Netwerkdiensten SARA e-science support E-science support assists scientist to utilize high performance computational infrastructure Daily work ranges from programming software tools to consultancy is e-science projects. A team of 9 persons in 2008 Pieter van Beek TOPOS developer ToPoS |

SARA Reken- en Netwerkdiensten Main Grid projects BioAssist - NBIC (Netherlands BioInformatics Centre) Bioinformatics Life Science Grid Grid for life sciences Big Grid Grid infrastructure and e-science LOFAR satellite dishes in Netherlands and Germany ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Life Science Grid Various clusters in The Netherlands for bioinformatics computing. Clusters can be used separately or by using gLite middleware. Open to all Life Scientists in the Netherlands Support on various levels provided by SARA

SARA Reken- en Netwerkdiensten Users experiences with gLite Overhead for starting jobs is considerable Determining the best chunk size is difficult. Too small -> large overhead Too large -> timeouts and throughput problems. Resource brokering is far from optimal Jobs often fail and users create their own tools for administrative tasks ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Resource Brokering ToPoS | 23 October 2008 Submitted jobs are sent to a CE immediately. When another CE becomes available, you won't use it automatically

SARA Reken- en Netwerkdiensten Failing Jobs (1) Common experiences: Sorry, an Incomprehensible Error occurred Your VOMS Credential has expired What Job? Success! (but there’s no output) Failure! (but it ran just fine) Out of Wall-time (but no CPU-time?) A lot of “monitoring and resubmission” software is created again and again by many users. ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Failing Jobs (2) A real world example: 27,000 jobs duration: approx. 4 hrs approx. 280 WNs Theoretical duration: 16 days But with a success rate of 70% … Approx. 9 resubmissions “Practical” duration: >2 months ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Pilot Jobs ToPoS | 23 October 2008 “Normal” jobs Pilot jobs

SARA Reken- en Netwerkdiensten Simplest possible solution: Topos I An online counter, like a “page views” counter Numbers are “leased” for some period Leases must be renewed Interfaced with HTTP (REST web service) Can be used with any HTTP client (wget, browsers) As little security as possible ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Pilot job flow ToPoS | 23 October 2008 Pilot job affirm token use affirm token use Get unused token Get unused token Submit Pilot job with token Running pilot job Execute token task Execute token task Finished ? Delete token Delete token no yes

SARA Reken- en Netwerkdiensten Advantages Simple design and use Using HTTP REST Automatic resubmissions Less overhead for large number of jobs. One pilot job can execute several tasks in sequence. Improved scheduling Easy job administration by querying Token Pool Server. Progress Fail rate ToPoS | 23 October 2008

SARA Reken- en Netwerkdiensten Topos I screenshots ToPoS | 14 November 2008

SARA Reken- en Netwerkdiensten User experiences First users are biologists. A large number of sequence aligments Weeks of execution time Originally a high failure rate TOPOS improved this situation considerably Easily scripted by using CURL Progress could be monitored by a web browser …without Grid certificate ToPoS |

SARA Reken- en Netwerkdiensten Topos 2.x Interfaced by WebDAV i.o. HTTP Tokens are files, i.e. they have identity content mime-type properties Token pools are directories Tokens can be moved between directories Allows users to build pipelines and workflows (high-level colored Petri nets) ToPoS |

SARA Reken- en Netwerkdiensten Topos 2 screenshot ToPoS |

SARA Reken- en Netwerkdiensten “Portfolio” SciaGrid Collaboration between SRON, KNMI, NIKHEF and SARA Website where users can select  satellite data (Sciamachy)  data processors Arnold Kuzniar and Jack Leunissen (WUR) BLAST protein sequence alignment Bas Dutilh (CMBI) HAMMER sequence alignment (?) Jan Bot (TUD) ToPoS |

SARA Reken- en Netwerkdiensten Future directions Documentation ATOM/RSS instead of WEBDAV Back to numbers instead of files TODO ToPoS |

SARA Reken- en Netwerkdiensten ToPoS | 23 October 2008