© 2002 Barton P. MillerMarch 4, 2001Tool Dæmon Protocol The Tool Dæmon Protocol: Using Monitoring Tools on Remote Applications Barton P. Miller Computer Sciences Department University of Wisconsin Madison, Wisconsin USA

– 2 –© 2002 Barton P. Miller Tool Daemon Protocol The Challenge Consider remote process management environments: qCondor qMPI Portable MPI (such as MPICH) Vendor provided MPI (such as IBM, Compaq, Sun) qGlobus Each of these environments needs to monitor and control the state of its application processes.

– 3 –© 2002 Barton P. Miller Tool Daemon Protocol Typical Process Manager Process manger: Starts the remote job Monitors its status Controls the job Sets up file I/O Sets up standard I/O Remote Process Manager monitor/ control Application Process Application Process Remote Host

– 4 –© 2002 Barton P. Miller Tool Daemon Protocol Typical Process Manager Remote Process Manager monitor/ control Application Process Application Process Remote Host Tool Dæmon Process ? ? The run-time tool? Also may want to start process (or attach to it) Also needs to monitors its status Also may want to control the job Needs to communicate with its front-end.

– 5 –© 2002 Barton P. Miller Tool Daemon Protocol Typical Process Manager Remote Process Manager monitor/ control Application Process Application Process Remote Host ? ? Tool Dæmon Process So, who wins?

– 6 –© 2002 Barton P. Miller Tool Daemon Protocol Typical Process Manager Remote Process Manager monitor/ control Application Process Application Process Remote Host Local Host ? ? Tool Front-End Process Tool Dæmon Process

– 7 –© 2002 Barton P. Miller Tool Daemon Protocol Current State of Affairs qEach process manager starts and controls processes in its own way. E.g., even within MPI: IBM POE MPI, SGI Origin MPI and MPICH all work differently. MPI has no standard process control! qSpecialized cases of a specific tool working with a specific environment e.g., TotalView debugger working with MPICH. qThe result is an m  n combination of m process managers and n tools. Bottom line: need a standard interface for process managers and tools to coexist: the Tool Dæmon Protocol (TDP).

– 8 –© 2002 Barton P. Miller Tool Daemon Protocol The Basic TDP Steps 1.Create, but don’t start, new application process. 2.If necessary, create tool daemon process. 3.Pass basic information to tool daemon: e.g., 1.Application PID. 2.Front-end host/port number. 3.Standard I/O host/port number. 4.Tool daemon processes application: 1.For a debugger, read symbols 2.For Paradyn/dyninst, parse the executable. 5.Start the application process 6.Respond to changes in the application state. 7.Respond to changes in the tool daemon’s state.

– 9 –© 2002 Barton P. Miller Tool Daemon Protocol Step 1a: Request Process Creation Remote Host Local Host Tool Front-End Process Remote Process Manager Create process

– 10 –© 2002 Barton P. Miller Tool Daemon Protocol Step 1b: Create Application Process Application Process Remote Host Local Host Tool Front-End Process Remote Process Manager

– 11 –© 2002 Barton P. Miller Tool Daemon Protocol Step 2: Create Tool Dæmon Application Process Remote Host Local Host Tool Dæmon Process Remote Process Manager Tool Front-End Process

– 12 –© 2002 Barton P. Miller Tool Daemon Protocol Step 3: Pass Info to Tool Dæmon Application Process Remote Host Local Host Remote Process Manager Tool Front-End Process PID, host/port pairs Tool Dæmon Process

– 13 –© 2002 Barton P. Miller Tool Daemon Protocol Step 4: Process the Application Remote Host Local Host Remote Process Manager Tool Front-End Process Tool Dæmon Process Application Process

– 14 –© 2002 Barton P. Miller Tool Daemon Protocol Step 5: Start the Application Remote Host Local Host Remote Process Manager Tool Front-End Process Tool Dæmon Process Application Process

– 15 –© 2002 Barton P. Miller Tool Daemon Protocol Interesting Challenges 1.Return codes and state changes? 2.Who control standard input and output? 3.Private networks and firewalls: 1.Tool dæmon communicating to front-end. 2.Application process sending standard I/O.

– 16 –© 2002 Barton P. Miller Tool Daemon Protocol Challenge: Firewalls and Private Nets Remote Host Local Host Remote Process Manager Tool Front-End Process Tool Dæmon Process Application Process Firewall X

– 17 –© 2002 Barton P. Miller Tool Daemon Protocol Challenge: Firewalls and Private Nets Remote Host Local Host Remote Process Manager Tool Front-End Process Tool Dæmon Process Application Process Firewall Comm Proxy

– 18 –© 2002 Barton P. Miller Tool Daemon Protocol Challenge: Firewalls and Private Nets qWhen tool daemon is started, pass in the host/port number of its front-end process. qIf there is a communication proxy, then: Tool daemon will receive host/port of the proxy, so daemon connects to proxy. Proxy will connect to the tool front-end, mapping the host/port (similar to NAT). qApplication connecting to console for standard I/O works the same way.

– 19 –© 2002 Barton P. Miller Tool Daemon Protocol The Condor/Paradyn Scenario Condor Starter monitor/ control Application Process Application Process Remote Host Local Host Paradyn Front-End Paradyn Dæmon

– 20 –© 2002 Barton P. Miller Tool Daemon Protocol The Path Forward qProduce a prototype implementation to expose technical challenges: Paradyn running under Condor Ana Cortes and Miquel Senar (UAB/UW) A presentation tomorrow about the first steps. qGoal is to produce a standard set of libraries for process managers and tool daemons. qInvolve a wider community in this standards effort Initially: ANL (Gropp and Lusk), Etnus (Cownie and Delsignore), Globus (Kesselman)

– 21 –© 2002 Barton P. Miller Tool Daemon Protocol The Path Forward qThis is just the beginning of an important effort qYour input and participation is welcome and encouraged. This project is supported by the hard work and advice of the Condor and Paradyn teams!