AppLeS, NWS and the IPG Fran Berman UCSD and NPACI Rich Wolski UCSD, U. Tenn. and NPACI This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab Type in action items as they come up Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered.

AppLeS and the IPG Usability, Integration development of basic IPG infrastructure Development of persistent IPG testbed Performance “IPG - aware” programming Short-termMedium-termLong-term Application scheduling Resource scheduling Throughput scheduling Multi-scheduling Resource economy Integration of schedulers and other tools, performance interfaces Experience with Pilot IPG Development of prototype performance-oriented applications Development of necessary research

A Model for the Future Adaptation is key to the ultimate IPG program development and execution environment. Exchange of performance information fundamental to the success of IPG applications PSEPSE Config. object program whole program compiler Source appli- cation libraries Realtime perf monitor Dynamic optimizer Grid runtime system negotiation Software components Service negotiator Scheduler Performance feedback Perf problem Grid Application Development System (GrADS)

Why Application Schedulers? Application performance can conflict with performance goals of other system components Goal of application scheduler is to prioritize performance of the application over other system components

Agent-based Application Scheduling Sensor Interface Reporting Interface Forecaster Model NWS User Prefs App Perf Model Planner Resource Selector Application Act. IPG /Globus infrastructure NWS (Wolski) AppLeS (Berman and Wolski)

Performance Prediction Given monitored bandwidth data, what will happen next?

NWS Predictions Monitored data provides a snapshot of what has happened. What we really want to know is: What will happen?

Monitoring vs. Prediction Last value not always the best predictor Hard to develop accurate forecasting models -- why not use all feasible models? Monitored data

Do AppLeS and NWS Improve Application Performance? Good results with many applications including –SARA AppLeS –CompLib AppLeS –Jacobi2D AppLeS AppLeS/NWS applications demonstrate that –prediction is possible in high-variance environments –adaptivity can improve performance

SARA AppLeS SARA = Synthetic Apperture Radar Atlas –application developed at JPL and SDSC Goal: Process radar images from distributed database for user’s desired image AppLeS focuses on resource selection problem

... Compute Servers Data Servers Client SARA Experiments

CompLib AppLeS Problem: Find the best matches between two gene sequence libraries Apply FASTA algorithm to all sequence pairs to determine similarity Developed for DOCT testbed sequence library

CompLib Experiments

Jacobi2D AppLeS Important component of many scientific applications Time-balancing used to achieve minimal execution time Scheduler solves time- balancing equations for Area

Jacobi2D Experiments Comparison of AppLeS with and without NWS info, and load-balancing

Applying AppLeS/NWS Methodology to the IPG AppLeS/NWS methodology can be used to develop performance-efficient IPG applications IPG FY99 projects leverage FY98 project and previous AppLeS/NWS development and research

IPG FY99 Project: A “Parameter Sweep” Template INS2D representative of larger class of critical NASA applications AppLeS parameter sweep template will build on INS2D model and experiments to target larger class of applications and platforms Template will serve as a prototype IPG PSE workbench tool AppLe S API Resources App- specific case gen. Exp Act Sched. Act Exp

AppLeS Project Plan FY99 (Berman,UCSD) Expand INS2D AppLeS –to NASA IPG testbed –to include batch systems –to target Globus Development of Parameter Sweep AppLeS template Goal: To provide framework for improving turnaround time of parameter study component of complex AES applications AppLeS scheduling agents prototype autonomous agent technology for IPG Requires development of strategy for scheduling in mixed batch and interactive environments Project Personnel: Berman, Casanova (UCSD) Collaborators: Wolski (U. Tenn.), Kesselman (ISI/USC)

NWS Project Plan FY99 (Wolski, U. Tenn.) Enhance the NWS to support AppLeS parameter sweep template in NASA Globus environment –NWS API for parameter sweep template –integration with Globus Integrate NWS with IPG and Globus application performance monitoring tools –use NWS performance techniques to predict application performance dynamically Investigate strategies for monitoring and forecasting batch system performance –queue wait times in the presence of user priorities, etc. Project Personnel: Wolski (U. Tenn) Collaborators: Berman (UCSD), Moore (SDSC), Kesselman (ISI/USC)

Possible Additional IPG Projects AppLeS/NWS-enhanced Storage Resource Broker Project: Enhance SRB performance through agent-based scheduling Project Personnel: Berman, Wolski Collaborator: Moore AppLeS/NWS-enhanced NetSolve over Globus Project: Improve scheduling component of NetSolve using AppLeS/NWS techniques, deploy on Globus IPG platform Project Personnel: Berman, Wolski, Casanova, Dongarra Collaborator: Kesselman

Possible Additional IPG Projects AppLeS/NWS Applications on Condor Project: Develop AppLeS application which can achieve performance in the Condor environment; integrate Condor and NWS information; leverage Condor/Globus integration Project Personnel: Berman, Wolski Collaborator: Livny, Kesselman

Project Information AppLeS Home Page: hpcl/apples.html NWS Home Page:

Project Information NWS Home Page: AppLeS + NWS Project Personnel –Francine Berman –Rich Wolski –Walfredo Cirne –Marcio Faerman –Jaime Frey –Jim Hayes –Graziano Obertelli AppLeS Home Page: cse.ucsd.edu/groups/ hpcl/apples.html –Jenny Schopf –Gary Shao –Neil Spring –Shava Smallen –Alan Su –Dmitrii Zagorodnov