1. GridLab Applications Gabrielle Allen, Zhou Lei, Archit Kulshrestha Center for Computation & Technology, LSU

2. Modeling Black Holes Einstein’s equations of General Relativity: G ab = 0 One of the most complex set of equations in physics –Mixed hyperbolic-elliptic initial value problem –3 Dimensional, multiple scales –12 or more equations, with thousands of terms Require diverse distributed collaborations & many large scale simulations: –Parallelization, optimization –Parallel IO, data management, viz –Checkpointing, steering, interaction t=100 t=0 Cactuscode.org

3. Application Scenarios Intelligent Automatic Parameter Surveys Dynamic Migration: faster/cheaper/bigger machine Multiple Universe: create job clone to investigate steered parameter Automatic Component Discovery and Loading: Needs of job change Automatic Convergence Testing Look Ahead: Spawn off and run coarser resolution to predict likely future Spawn Independent/Asynchronous Tasks –send to cheaper machine, main simulation carries on Routine Profiling –best machine/queue, choose resolution parameters based on queue Dynamic Load Balancing and Optimization: inhomogeneous loads, multiple grids, network and cache hierarchies.

4. Application Scenarios Automatically record details about simulations and results. Store locations of output data files, along with searchable metadata. Ontologies for describing application area: use for meta data, connection to journal publications etc. All done in a community accessible way

5. SCOOP: SURA Coastal Ocean Observing Program (twiki.sura.org) –NOAA/ONR Data coming in from sensors all over Gulf: Realtime Operational Grid Feeds in to models on LSU SuperMike and other Grid sites Indication of strengthening storm changing direction: Must Act! Multiple simulations, expertise, and data connected together: FL, LA, notification, steering sensors Early warning, preparation: –Insurance industry, petroleum industry, local economy all save millions, lives saved Hurricanes on the Grid LSU, GoMOOS, Texas A&M, UaH, UNC/MCNC, U. Florida, U. Miami, VIMS

6. Ubiquitous Computing and Monitoring System (UCoMS) $2.4M DOE/BOR project ULL, LSU, Southern, Landmark –Petroleum Engineering, CS, ULL –Deploy sensor networks across Gulf –Data collected to provide input to simulations, tasks farmed out –Results collected (transmitted back) http://www.ucoms.org

7. LRZ Dynamic Grid Computing

8. Application Tools for Grids Application developers need the right kinds of tools and environments to use Grids –APIs, toolkits, debuggers, … Focused on the application requirements (MPI, Fortran, legacy, ….) Hide the complexities of the Grid Work in a familiar way (avoid new paradigms) –Functioning and reliable heterogeneous testbed environments for developing and testing –Support new and dynamic application scenarios, as generic as possible.

9. GridLab Project EU Funded (5th Framework): Jan 2002 –Partners: PSNC, AEI, ZIB, MASARYK, VU, SZTAKI, ISUFI, Cardiff, NTUA, Chicago, ISI, Wisconsin, Sun, Compaq –GridSphere Portal –Applications Numerical Relativity (Cactus) Gravitational Waves (Triana) Mobile Users –Europe-wide Test Bed (Linux, SGI, Hitachi, …) –Grid Application Toolkit (GAT) Range of Grid Services http://www.gridlab.org

10. Grid Application Toolkit (GAT) Application oriented access to Grid through a standard API –GATFile_Move –GATResource_Find –GATFile_Logical –Independent of Grid infrastructure and available services. C, C++, Java, Python, [Perl/Fortran] All platforms: Unix, Windows API driver for GGF SAGA-RG

11. GAT Motivation Use of Grids by applications is lagging behind the development of Grid technologies, big jump from prototypes and demonstrations to real production use of Grids needed by applications. Problems: –Missing or immature grid services. –Changing environment, new software versions. –Different and evolving interfaces to the “grid”. –“Functioning” Grid is needed to develop applications. –Interfaces are not simple for scientific application developers.

12. Monitoring Resource Management Information Security Data Management GLOBUS Application Manager Logging Notification Migration Profiling SOAPWSDLCorbaOGSAOther Other Grid Infrastructure? Application “Copy my file from there to there..” Example: Remote File Copying

13. Copy a File: Globus GASS int RemoteFile::GetFile (char const* source, if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP || char const* target) { source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_url_t source_url; globus_ftp_client_operationattr_init (&source_ftp_attr); globus_io_handle_t dest_io_handle; globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, globus_ftp_client_operationattr_t source_ftp_attr; &source_ftp_attr); globus_result_t result; } globus_gass_transfer_requestattr_t source_gass_attr; else { globus_gass_copy_attr_t source_gass_copy_attr; globus_gass_transfer_requestattr_init (&source_gass_attr, globus_gass_copy_handle_t gass_copy_handle; source_url.scheme); globus_gass_copy_handleattr_t gass_copy_handleattr; globus_gass_copy_attr_set_gass(&source_gass_copy_attr, globus_ftp_client_handleattr_t ftp_handleattr; &source_gass_attr); globus_io_attr_t io_attr; } int output_file = -1; output_file = globus_libc_open ((char*) target, if ( globus_url_parse (source_URL, &source_url) != GLOBUS_SUCCESS ) { O_WRONLY | O_TRUNC | O_CREAT, printf ("can not parse source_URL \"%s\"\n", source_URL); S_IRUSR | S_IWUSR | S_IRGRP | return (-1); S_IWGRP); } if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); if ( source_url.scheme_type != GLOBUS_URL_SCHEME_GSIFTP && return (-1); source_url.scheme_type != GLOBUS_URL_SCHEME_FTP && } source_url.scheme_type != GLOBUS_URL_SCHEME_HTTP && /* convert stdout to be a globus_io_handle */ source_url.scheme_type != GLOBUS_URL_SCHEME_HTTPS ) { if ( globus_io_file_posix_convert (output_file, 0, printf ("can not copy from %s - wrong prot\n", source_URL); &dest_io_handle) return (-1); != GLOBUS_SUCCESS) { } printf ("Error converting the file handle\n"); globus_gass_copy_handleattr_init (&gass_copy_handleattr); return (-1); globus_gass_copy_attr_init (&source_gass_copy_attr); } globus_ftp_client_handleattr_init (&ftp_handleattr); result = globus_gass_copy_register_url_to_handle ( globus_io_fileattr_init (&io_attr); &gass_copy_handle, (char*)source_URL, &source_gass_copy_attr, &dest_io_handle, globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); my_callback, NULL); &io_attr); if ( result != GLOBUS_SUCCESS ) { globus_gass_copy_handleattr_set_ftp_attr printf ("error: %s\n", globus_object_printable_to_string (&gass_copy_handleattr, (globus_error_get (result))); &ftp_handleattr); return (-1); globus_gass_copy_handle_init (&gass_copy_handle, } &gass_copy_handleattr); globus_url_destroy (&source_url); return (0); }

14. GAT Solution API layer between applications and the grid infrastructure: –Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs –Insulate against Rapid evolution of grid infrastructure State of Grid deployment –Choose between different grid infrastructures –Application developers use and develop for the grid independent of the state of deployment of the grid infrastructure –Service developers can make their software available to many different applications.

15. Copy a File: GAT/C++ #include GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS; }

16. Example: Remote File Copy GAT:File.Copy scpcpgsiscp sftp gridftp wgetRFT GridLab Project Specific GridJITnewest Application

17. Grid Application Toolkit Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services Implements the GAT-API –Used by applications GAT Adaptors –Connect to capabilities/services –Implements well defined CPI (mirrors GAT-API) –Interchangeable adaptors can be loaded/switched at runtime GAT Engine –Provides the function bindings for the GAT-API

18. GridLab Architecture

19. GAT API Scope Files Resources Events Information exchange Utility classes (error handling, security, preferences)

20. GAT Adaptors Default Adaptors: –Basic functionality which will work on a single isolated machine (e.g. cp, fork/exec) GridLab Adaptors: –Connect to services developed through the GridLab project (GRMS, Mercury, Delphoi, iGrid) Globus Adaptors: –Core Globus functionality: GRAM, MDS, GT- RLS, GridFTP Others: DRMAA, Condor, SGE, gsissh, …

21. GGF SAGA-RG Simple API for Grid Applications –Global Grid Forum Research Group –Motivated by GAT-API and other efforts to develop a real common standard DRMAA, Java-COG, GAT-API, … –Meeting at LSU Dec04 will provide an initial draft A standard API, implemented and supported by vendors, will greatly advance the deployment and adaoption of Grid applications

22. Simple Examples Many short examples and “one liners” included in GAT distribution: –Read from a remote file –Read from a logical file –Move a file –Spawn a subtask to a remote machine –Information about remote/logical file (e.g. file size, permissions)

23. Example: Spawn a Subtask GAT::Table sdt; sdt.add ("location", "/bin/date"); GAT::Table hdt; hdt.add ("machine.type", "i686"); GAT::SoftwareDescription sd (sdt); GAT::HardwareResourceDescription hrd (hdt); GAT::JobDescription jd (context, sd, hrd); GAT::ResourceBroker rb (context, prefs); GAT::Job j = rb.submit (jd);

24. TFM Task Farming on the Grid Task farming infrastructure implemented in Cactus GAT used for starting remote TFMs Designed for the Grid Tasks can be Anything (MPI, Single proc)

25. Black Holes on the Grid Task farm small Cactus black hole simulations across GGTC Grid Parameter survey: black hole corotation parameter Results steer a large production black hole simulation

26. Job Migration

27. Migration of Legacy Apps Using e.g. wrapper scripts the GAT can be used to help Grid-enable “legacy” applications. Use a GAT wrapper to start Legacy App Send a checkpoint request to the GAT Wrapper Legacy Application writes checkpoint file GAT Wrapper Exits – Kills Legacy App GAT Wrapper registers checkpoint file to logical file GRMS Replicates logical File on target Host GRMS starts the GAT Wrapper on target host Legacy Application is started within GAT Wrapper

28. Notification and Information GridSphere Portal SMS Server Mail Server “The Grid” Replica Catalog

29. GridSphere Portal Portlet based: JSR 168 Collaboration focal point Interact, share data Start, monitor remote jobs Move/browse files Track and monitor announced jobs Application specific portlets –Cactus, UCoMS, SCOOP

30. Portlets JSR-168 lays the foundation for a new open-standard for Web portal development frameworks. Multiple portlets can be composed in a portal page. Portlets define how to construct and deliver Web content as modular components. Portlets can be “maximized” or “minimized”. Portlets support various modes (View, Edit, Help, Configure) Users can subscribe to portlets.

31. GridSphere Feature List Portlets compatible with IBM's WebSphere 4.2. Support for easy integration of "third-party portlets" Higher-level model for building complex portlets. Customized layouts. Built-in support for Role Based Access Control (RBAC) separating users into guests, users, admins and super users. Sophisticated portlet service model. Documentation (DocBook) HTML/PDF guides and tutorials. GridSphere core portlets offer base functionality including login, logout, user and access control management. Grid Portlets add general Grid Capabilities.

32. So what does it look like?

33. Grid Portlets Resource Registry Portlet (Grid/Administration Tab) –Administrators can customize for particular VO Resource Browser Portlet (Grid/Resources Tab) –Resource information from MDS/iGrid etc Credential Retrieval Portlet (Grid/Credentials Tab) –MyProxy Job Submission Portlet (Grid/Jobs Tab) –GRAM, GRMS File Browser Portlet (Grid/Files Tab) –GridFTP …and many more coming… Implements generic Grid Portlet API (similar to GAT)

34. Resource Browsing

35. Job Submission Portlet

36. Job History & Output

37. Links GridLab: –http://www.gridlab.org Grid Application Toolkit –http://www.gridlab.org/GAT GridSphere: –http://www.gridsphere.org

38. Mardi Gras Conference Frontiers of Grid Applications and Technologies Louisiana State University Baton Rouge February 3rd to 5th, 2005 http://www.mardigrasconference.org