1. EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Integration of Astro-WISE with Grid storage Fokke Dijkstra, Kor Begeman, Andrey Belikov, Danny Boxhoorn, Edwin Valentijn, Willem-Jan Vriend, Zheng Zhao Donald Smits Center for Information technology / Kapteyn Astronomical Institute University of Groningen EGEE Conference Barcelona, 24 September 2009

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 2 Astro-WISE information system – fully datacentric All data beyond pixel data is Metadata all pixel data data servers all Metadata database compute clusters / GRIDs all I/O to db all components scalable all components EU distributed

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 3 Astro-WISE paradigm “Classical” paradigm Forward chaining Target processing - Awe Backward chaining waterfall model TIER architecture User hunts upstream driven by input raw data Driven by query of user Process in pipeline workflow Process in bits and pieces on the fly Backward chaining Operators push data User pulls data Results in releases Provide information system Static archives – publish Dynamic archives –publish Internet Raw data - obsolete Raw data is sacred

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 4 Target diagram

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 5 VST - Virtual Survey Telescope

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 6 The Avalanche on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Python Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool” Administration for parallel processing -compute GRID  Connect to EGEE - Grid Global solutions  astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 7 Connection to EGEE Lofar long term archive uses Grid resources –International user community –Support from Big Grid (Dutch NGI) Grid credentials handled using external tools –MyProxy upload tool (http://www.grid- support.ac.uk/content/view/396/173/)http://www.grid- support.ac.uk/content/view/396/173/ –jLite (http://code.google.com/p/jlite/)http://code.google.com/p/jlite/ –Python myproxy client (http://proj.badc.rl.ac.uk/ndg/wiki/Security/MyProxyClient)http://proj.badc.rl.ac.uk/ndg/wiki/Security/MyProxyClient Job submission described at Istanbul conference –Distributed Processing Unit on gLite UI using delegated proxy

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 8 Connection to Grid storage Requirements –Support multiple protocols  Astro-WISE data servers  TarGet GPFS storage  SRM Grid storage  ? –Store entries to files in Astro-WISE db –Allow for and keep track of replicas –Access rights on metadata may differ from those on file location information –Integrate with Python environment  Using lightweight Grid clients

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 9 Implementation Grid –Separate Storage table –Object oriented approach supports multiple protocols –SRM URIs stored for Grid –Interfaced to proxy handling routines –Retrieve and store operations  Interfaced to dCache SRM client (java)

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 10 Example session $ awe Python 2.5.4 (r254:67916, Aug 26 2009, 13:15:31) [GCC 4.3.3] on linux2 Welcome to the LOFAR Astro-WISE Environment Importing Astro-WISE packages. Please wait... [rc-440.rc.rug.nl] 16:57:38 - Using Lofar LTA Storage awe> from common.external.jlite import proxy_init awe> proxy_init() 0 awe> from common.database.DataObject import DataObject awe> myobject=DataObject() awe> myobject.pathname="/home/fokke/test.txt" awe> myobject.store() [rc-440.rc.rug.nl] 17:13:17 - Storing /home/fokke/test.txt as srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/2009-09-16/test.txt [rc-440.rc.rug.nl] 17:13:31 - Warning; could not determine remote hash value: srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/2009-09-16/test.txt [rc-440.rc.rug.nl] 17:13:31 - FileObject.check_preconditions awe> myobject.commit() awe> myobject.retrieve() [rc-440.rc.rug.nl] 17:13:44 - Retrieving srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/2009-09-16/test.txt, using protocol srm awe>

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 11 Lofar Long Term Archive

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 12 Conclusions Successful integration of Astro-WISE and EGEE Storage and processing integrated –Portable implementation  jLite  dCache SRM Power of Astro-WISE available for Lofar user community Future work on improving robustness Plan for demo at next user forum First Lofar observation of Cygnus A with three stations (Copyright: ASTRON)

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 13 Web services Target processor

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona 2009 14 Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd53‘) and (ReducedScienceFrame.filter == ‘#841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat (dependency) Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)