INFSO-RI Enabling Grids for E-sciencE Project Gridification: the UNOSAT experience Patricia Méndez Lorenzo CERN (IT-PSS/ED) CERN, 1st March 2006 EGEE USER FORUM

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Outlook ◘ UNOSAT Presentation ◘ Collaboration with CERN ◘ UNOSAT and GRID ◘ Production of UNOSAT inside the Grid environment ◘ Future Plans

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo UNOSAT Presentation UNOSAT is a United Nations Initiative ◘ Objectives ➸ Provide the humanitarian community with access to satellite imagery and Geographic Information System services ► Reduce disasters and plan sustainable development ➸ Ensure cost-effective and timely products ◘ Core Services ➸ Humanitarian Mapping ➸ Image Processing VEGETATION – 1 Km IKONOS – 1m

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo UNOSAT Service Chain Data suppliers Ground station USER UNOSAT Central Unit WWW    a) Detailed satellite maps of the area are required by UNOSAT Satellite responds the requirement with radar and optic techniques b) Satellites send raw data back to Earth c) Advanced data processing performed d) Data relayed by UNOSAT e) Additional processing and download in the web page of UNOSAT

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Relief Projects of UNOSAT ◘ Case Study: Indian Ocean Tsunami Relief and Development ◘ 29 th Dec 2004: First Map distributed online to field users ➸ 14 th Jan 2005: Imagery Bank online: ► 100 Tsunami-related maps (pre and post) ► 670 raw satellite images ➸ January: 200,000 tsunami maps downloaded in total ◘ After the Tsunami catastrophe ➸ Pakistan ➸ Filipinas ◘ In both cases maps are hosted at CERN ◘ Projects founded by European partners as GMES (Global Monitoring Environment and Security)

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo First step: UNOSAT and CERN ◘ UNOSAT is CERN partner since 2002 ◘ CERN supports them with network facilities, with computer infrastructure and with human (support) resources ◘ Asian Tsunami Example: ➸ Central Web Services at CERN under considerable strain ➸ Availability of the UNOSAT data at any time ◘ UNOSAT provides the users with a web interface able to find the files of the images by clicking on the earth images ➸ Attractive method, easy to use ◘ Something similar to do with the GRID ➸ One of our goals inside the UNOSAT-GRID Collaboration

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo One step further: GRID ◘ Potential Bottlenecks: ➸ UNOSAT beginning to suffer from limited capacity and processing power ➸ Multiple satellites being launched ➸ Larger and larger storage capacity needed ◘ In summer 2005 we have provided a whole structure at CERN for UNOSAT ➸ UNOSAT Virtual Organization (VO) ➸ 3.5TB in CASTOR ➸ Computing Elements, Resource Brokers ➸ Collaboration with ARDA group ➸ AFS area of 5GB ◘ We have run some UNOSAT tests (images compression) inside the GRID environment (quite successful) ◘ The framework developed for in principle for Geant4 (See Alberto Ribon’s presentation [49]) has been adapted for UNOSAT needs We have provided the whole GRID infrastructure At CERN

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo UNOSAT Production inside GRID ◘ UNOSAT provided us with a set of images for testing ◘ Associated to each image a metadata file was included File name, directory path, geographical coordinates ◘ Steps : STORAGE LEVEL ➸ Copy and registration of the images in ► Use of the LFC Catalog ➸ Parse the metadata files to extract the different metadata ➸ Use of the AMGA tool to parse metadata to location of the files COMPUTING LEVEL ➸ Use of compression tools to compress images inside LCG resources ➸ Use of the general submission tool adapted to UNOSAT needs

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo A GRID Metadata Catalogue ◘ LFC Catalogue ➸ Mapping of LFN to PFN ◘ UNOSAT requires ➸ User will give as input data certain coordinates ➸ As output, he wants the PFN for downloading ◘ The ARDA Group assists us setting up the AMGA tool for UNOSAT Oracle DB ARDA APP LFC CASTOR SRM Metadata (x,y,z) LFN PFN

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Tools developed for new Gridifications Generation of a general framework consisting of 2 major tools: 1. Tool to perform the automatic job submission 2. Tool to retrieve and handle the corresponding output 1. Automatic job submission ◘ Given an user’s jdl this tool performs the following actions: ➸ It lists all sites able to run the jdl provided by the user ➸ It creates automatically a jdl file based on that provided by the user ➸ It submits the just created jdl containing the user application(s) ➸ Moreover it creates a subdirectory (defined by the user) containing a list of the sites where the jobs have been submitted, the corresponding jdls and the jobs IDs

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Tools developed for new Gridifications ◘ Additional Features: ➸ The user can define the queues where the jobs are submitted. These queues are checked to see whether it fixes the job requirements. ➸ Requested LFN files can be included. The corresponding TURLs are searched and included in a file passed in the InputSandbox to the WN ◘ Applications ➸ This tool has been used for the 1st and the 2nd phases of the production: software installation and production ◘ Usage:./submitter_general -vo geant4 -jdl jdlexample -jobfile G4_PROD -data /grid/geant4/production_software

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Tools developed for new Gridifications 2. Retrieve and handle of the outputs ➸ The 2 nd tool checks the status of the jobs from the job IDs included in the directory given by the user ◘ Usage:./get_output -jobfile G4_PROD -dest G4_PROD/outputs ◘ Output The job run in ramses.dcic.ups.es:2119/jobmanager-torque-dteam is in status: Scheduled The job run in grid01.phy.ncu.edu.tw:2119/jobmanager-torque-dteam is in status: running The job run in scaic10.scai.frauhofer.de:2119/jobmanager-torque-dteam is in status: over

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Tools developed for new Gridifications ◘ Additional Features: ➸ It is possible to visualize the outputs on the web ➸ A html report is provided showing the files decided by the user

Enabling Grids for E-sciencE INFSO-RI CERN, 1 st March 2006 Patricia Méndez Lorenzo Future Plans with UNOSAT ◘ Collaboration between UNOSAT, ARDA and GD ➸ 1(2) ARDA and 2 UNOSAT Students ➸ Still many discussions needed ➸ Support from other sites foreseen (x,y,z) GRID WORLD Application ◘ User can get the info in his laptop too ◘ Fundamental AMGA