NeSC Apps Workshop July 20 th, 2002 Customizable command line tools for Grids Ian Kelley + Gabrielle Allen Max Planck Institute for Gravitational Physics Golm, Germany GridTools:

NeSC Apps Workshop July 20 th, 2002 Introduction Simple command line tools (in Perl) for testing and performing operations across TestBeds. Motivation: –Working with 26 machines on the SC2001 testbed –Tools to help us get our physics users onto the Grid –Playground for easily testing different scenarios before building them into portals/applications. Have been useful for us, so put them together and wrote some documentation. See also TeraGrid pages: –

NeSC Apps Workshop July 20 th, 2002 TestBeds What do we mean by “TestBed”? –My –My definition of a TestBed: “a collection of machines with some sort of coordinated infrastructure, that is used for a common purpose or by a specific group of users –We want to develop, deploy and test portal and application software –Ultimately: want real “users” to view TestBed as a single resource For me: –SC2001 (GGF Apps) TestBed –GridLab TestBed –AEI Relativity Group production machines –My –My personal TestBed

NeSC Apps Workshop July 20 th, 2002 SC2001 TestBed 26 Machines Very heterogeneous All sites worked to build towards a common setup (GRAM, GSI, Cactus, Portal, GIIS) At SC2001 showed a Cactus simulation dynamically spawning individual analysis tasks to all machines

NeSC Apps Workshop July 20 th, 2002 NumRel Production TestBed This is what we really want! For physicists to do physics! Hard work! Blue Horizon Lemieux Globus sr8000 Los Lobos Platinum Titan Psi Seaborg Origin

NeSC Apps Workshop July 20 th, 2002 (Some) TestBed Gripes … Software deployment not yet standard/stable Information not easy-to-find or up-to-date Different security/account policies (firewalls!!) Priorities mean things not always fixed quickly. Hard to get a global view of current state. Have trouble keeping track of changes Not everything works as expected Basically we need to work in a “research-like” environment, but the more we use it, the more “production-like” it will become …

NeSC Apps Workshop July 20 th, 2002 What We Want To Do Run different tests (gsi, gram, etc) on our TestBed to verify that things are working correctly. Easily get up-to-date global views of our testbeds. Log files for tracking history, stability, etc. Easily add and configure machines and tests. Construct and test more complex scenarios for applications Something that our end-users can also use!

NeSC Apps Workshop July 20 th, 2002 Higher Level Scenarios For example for our portal/applications we want to test feasibility/usefulness etc of –Remote code assembly and compilation –Repositories of executables –Things specific for Cactus: parameter files, thornlists –Data description archiving, selection, transfer –Visualisation –Design of user-orientated interfaces –User customisations –Collaborative/Group issues –Simulation announcing/steering/tracking These also require work on the applications !!

NeSC Apps Workshop July 20 th, 2002 GridTools Aims Give a wrapper around Globus tools that enables scripting capability to perform multiple tasks. Provide additional functionality such as a pseudo database for storing machine and configuration specific information. Modularization of functionality to allow for easy development of more complex programs.

NeSC Apps Workshop July 20 th, 2002 What You Get Basic scripts: –TestAuth –TestResources Report making –CreateTEXT –CreateHTML –CreateMAP Other stuff A Library: –GridTools.pm A Pseudo-database –grid.dat –(all the stuff we really want to get from e.g. MDS)

NeSC Apps Workshop July 20 th, 2002 TestAuth Output

NeSC Apps Workshop July 20 th, 2002 Current Tests for TestResources Authorize to Globus Gatekeeper Simple GRAM job submission Using GSIFTP to copy files Using GSISCP to copy files Testing GSISSH in batchmode Simple job run using GASS server Simple MPI job run using GASS server Using machine specific predefined RSLs to execute a simple job Very simple to add new tests.

NeSC Apps Workshop July 20 th, 2002 TestResources Output

NeSC Apps Workshop July 20 th, 2002 TestResources Output

NeSC Apps Workshop July 20 th, 2002 Extensibility GridTools.pm, a Perl module, contains many common functions that allow you to easily write additional scripts or modify the existing ones. –Such as execution of commands via fork() or using timeouts –Reading of machine configuration –User (text based) interfaces Could implement other useful functionality –Timing how long things take to complete –More advanced monitoring How often do different services go down on different machines –Querying of information servers to update local database, or visa-versa GridTools can be extended to perform more complicated tasks. –Such as real job submission Using RSL templates and compilation specific information –Distribution or aggregation of files and processes

NeSC Apps Workshop July 20 th, 2002 Conclusion GridTools can help you to run tests on a group of computers to provide you with a general overview of the status of your TestBed. Can be extended to include more complicated tasks such as job distribution and compilation. Obtain from CVS: –cvs –d login password: anon –cvs –d co GridTools Contact me for help/comments: