PTools Annual Meeting, Knoxville, TN, September 2002 The Tool Daemon Protocol: Defining the Interface Between Tools and Process Management Systems Paradyn Group Condor Group { Computer Sciences Department University of Wisconsin Madison, Wisconsin USA Ana Cortés Miquel A. Senar Departament d’Informàtica Universitat Autònoma de Barcelona Barcelona, Spain Presented by Philip C. Roth

2Tool Daemon Protocol The Current Situation Consider a job submitted to a process management system (e.g., Condor, PBS, Globus, MPICH’s MPD)—the process manager… …starts the job’s processes Sets up file I/O Sets up standard I/O …monitors process status …controls the job Process Manager Daemon monitor/ control Application Process Application Process

3Tool Daemon Protocol The Current Situation Next, consider a tool wanting to monitor the job. The tool… …also may want to start the processes (or attach to them) …also needs to monitors process status …also may want to control the job …also may want access to file I/O or standard I/O …needs to communicate with its front-end Process Manager Daemon monitor/ control Application Process Application Process Tool Daemon ? ?

4Tool Daemon Protocol The Current Situation So, who wins? Process Manager Daemon monitor/ control Application Process Application Process Tool Daemon ? ?

5Tool Daemon Protocol The Current Situation Process managers are many and varied E.g., IBM POE, SGI Origin MPI and MPICH all work differently Some process managers have support for specific tools E.g., MPICH support for TotalView debugger Heading for an m  n combination of m process managers and n tools Bottom line: need a standard interface for process managers and tools to coexist: The Tool Daemon Protocol (TDP)

6Tool Daemon Protocol TDP: The Tool Daemon Protocol Defines an API between process management system and tool processes for… 1.Creating processes 2.Controlling processes 3.Sharing information between processes Pilot implementation—trying out ideas to see what works

7Tool Daemon Protocol TDP Job Startup Sequence Execution Host Local Host Tool Front-End Process Manager Daemon Create job 1.Tool submits job request to process management system

8Tool Daemon Protocol TDP Job Startup Sequence Application Process Execution Host Local Host Tool Front-End Process Manager Daemon 2.Process manager creates application processes, leaving it suspended (“pause on exec”)

9Tool Daemon Protocol TDP Job Startup Sequence Application Process Execution Host Local Host Tool Daemon Process Manager Daemon Tool Front-End 3.PM daemon creates tool daemon process (if necessary) TDP

10Tool Daemon Protocol TDP Job Startup Sequence Application Process Execution Host Local Host Process Manager Daemon Tool Front-End Tool Daemon PID, host/port pairs 4.PM daemon passes information to tool daemon (e.g., process pid, front-end host/port, standard I/O host/port)

11Tool Daemon Protocol TDP Job Startup Sequence Execution Host Local Host Process Manager Daemon Tool Front-End Tool Daemon Application Process 5.Tool daemon examines the application process (e.g., parses symbols, discovers static call graph)

12Tool Daemon Protocol TDP Job Startup Sequence Execution Host Local Host Process Manager Daemon Tool Front-End Tool Daemon Application Process 6.App process is allowed to run

13Tool Daemon Protocol TDP Pilot Implementation Goals To try out TDP ideas and see what makes sense in real environment To collect informed suggestions for a standard The software Two well-established packages at U. Wisconsin- Madison Paradyn performance tool Condor resource management system

14Tool Daemon Protocol 1.Process startup 2.Notification of exited processes 3.Inter-process communication Mechanism Identification of information to be transferred Asynchronous notifications 4.Private networks and firewalls Tool daemon communicating to front-end Application process sending standard I/O Challenges

15Tool Daemon Protocol Challenge: Process Startup Most functionality already in place, but not in the right place Need to refactor process startup logic between process manager daemon and tool daemon Control handoff (process manager daemon to tool daemon) difficult under some OSs E.g., Linux—two scheduling race conditions between application process and tool daemon

16Tool Daemon Protocol Challenge: Exit Process Notification Want the starter to be aware if the app or tool daemon process exits Process exit notification (e.g., SIGCHLD to the parent under UNIX/Linux) paradynd App SIGCHLD starter SIGCHLD Parent of

17Tool Daemon Protocol Challenge: Exit Process Notification paradynd App starter SIGCHLD Parent of Parental relationships may change when tool daemon attaches E.g., Linux—daemon process becomes app process’ parent  On app process’ termination, SIGCHLD sent to paradynd, NOT to the Condor starter Parent of

18Tool Daemon Protocol Challenge: Exit Process Notification paradynd App starter SIGCHLD SIGCHLD delivered to Condor starter only if paradynd calls wait()  Condor must trust monitoring daemon or poll the application process’ state

19Tool Daemon Protocol Challenge: Information Transfer “Attribute Space” {name, value} pairs shared between processes Mainly, intra-host sharing between process manager daemon and tool daemon Also tool front-end, daemon sharing E.g., application PIDs for front end Basic idea from MPICH Not a Linda tuple space Not a global shared environment space

20Tool Daemon Protocol Attribute Space (Execution Host) Process Manager Daemon Tool Daemon Application Process PID=2473 FE_host=cham.cs.wisc.edu FE_port=7331 tdp_put(“PID”, “2473”) tdp_put(“FE_host”, “cham.cs.wisc.edu”) tdp_put(“FE_port”, “7331”)

21Tool Daemon Protocol Attribute Space (Execution Host) Process Manager Daemon Tool Daemon Application Process PID=2473 FE_host=cham.cs.wisc.edu FE_port=7331 tdp_get(“PID”) tdp_get(“FE_host”) tdp_get(“FE_port”)

22Tool Daemon Protocol Challenge: Asynchronous Notification Uses attribute space In process interested in event notification, register action tdp_register_notify(handle, event, action) In event-generating process, deliver event to attribute space tdp_put(event,value) Value available in action function

23Tool Daemon Protocol Challenge: Firewalls and Private Nets Remote Host Local Host Process Manager Daemon Tool Front-End Tool Daemon Application Process Firewall X

24Tool Daemon Protocol Challenge: Firewalls and Private Nets Remote Host Local Host Process Manager Daemon Tool Front-End Tool Daemon Application Process Firewall Comm Proxy

25Tool Daemon Protocol Status Pilot implementation nearly complete Paradyn with jobs submitted to Condor Linux 2.4 “Create process” model Condor “vanilla” and “MPI” universes Remaining work: library packaging, documentation Periodic planning meetings Paradyn (Miller) Condor (Livny) U. Barcelona (Cortés, Senar) TUM (Wismüller) U. Vienna (Fahringer) U. Tennessee (Moore) MPICH (Butler, Gropp, Lusk) Etnus (Cownie, Delsignore) Globus (Kesselman) HP/Compaq (Balle) Pallas (Vampir group)

26Tool Daemon Protocol The Path Forward Identify necessary information exchange between principals Complete design, implement attribute space as standalone package Get other tool builders, process management system builders involved Integrate TDP ideas into their systems to see what works

27Tool Daemon Protocol Summary TDP standardizes the interface between process management systems and tools API for tools and management systems Support libraries Distributed attribute space Avoids the propagation of tool- and process manager-specific interfaces Pilot implementation nearly complete

28Tool Daemon Protocol TDP: The Tool Daemon Protocol It is the early stages of this important effort— we want your participation! Draft report in progress—available for review and comments soon Web: Barton Miller Philip Roth Brandon Schendel Victor Zandy Miron Livny Todd Tannenbaum Derek Wright Ana Cortés Miquel A. Senar Pilot Implementation Team