EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Integration of Astro-WISE with Grid storage Fokke Dijkstra, Kor Begeman, Andrey Belikov, Danny Boxhoorn, Edwin Valentijn, Willem-Jan Vriend, Zheng Zhao Donald Smits Center for Information technology / Kapteyn Astronomical Institute University of Groningen EGEE Conference Barcelona, 24 September 2009

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Astro-WISE information system – fully datacentric All data beyond pixel data is Metadata all pixel data data servers all Metadata database compute clusters / GRIDs all I/O to db all components scalable all components EU distributed

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Astro-WISE paradigm “Classical” paradigm Forward chaining Target processing - Awe Backward chaining waterfall model TIER architecture User hunts upstream driven by input raw data Driven by query of user Process in pipeline workflow Process in bits and pieces on the fly Backward chaining Operators push data User pulls data Results in releases Provide information system Static archives – publish Dynamic archives –publish Internet Raw data - obsolete Raw data is sacred

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Target diagram

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona VST - Virtual Survey Telescope

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona The Avalanche on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Python Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool” Administration for parallel processing -compute GRID  Connect to EGEE - Grid Global solutions  astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Connection to EGEE Lofar long term archive uses Grid resources –International user community –Support from Big Grid (Dutch NGI) Grid credentials handled using external tools –MyProxy upload tool ( support.ac.uk/content/view/396/173/) support.ac.uk/content/view/396/173/ –jLite ( –Python myproxy client ( Job submission described at Istanbul conference –Distributed Processing Unit on gLite UI using delegated proxy

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Connection to Grid storage Requirements –Support multiple protocols  Astro-WISE data servers  TarGet GPFS storage  SRM Grid storage  ? –Store entries to files in Astro-WISE db –Allow for and keep track of replicas –Access rights on metadata may differ from those on file location information –Integrate with Python environment  Using lightweight Grid clients

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Implementation Grid –Separate Storage table –Object oriented approach supports multiple protocols –SRM URIs stored for Grid –Interfaced to proxy handling routines –Retrieve and store operations  Interfaced to dCache SRM client (java)

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Example session $ awe Python (r254:67916, Aug , 13:15:31) [GCC 4.3.3] on linux2 Welcome to the LOFAR Astro-WISE Environment Importing Astro-WISE packages. Please wait... [rc-440.rc.rug.nl] 16:57:38 - Using Lofar LTA Storage awe> from common.external.jlite import proxy_init awe> proxy_init() 0 awe> from common.database.DataObject import DataObject awe> myobject=DataObject() awe> myobject.pathname="/home/fokke/test.txt" awe> myobject.store() [rc-440.rc.rug.nl] 17:13:17 - Storing /home/fokke/test.txt as srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/ /test.txt [rc-440.rc.rug.nl] 17:13:31 - Warning; could not determine remote hash value: srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/ /test.txt [rc-440.rc.rug.nl] 17:13:31 - FileObject.check_preconditions awe> myobject.commit() awe> myobject.retrieve() [rc-440.rc.rug.nl] 17:13:44 - Retrieving srm://srm.grid.rug.nl:8444/omegac/awfdijkstra/ /test.txt, using protocol srm awe>

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Lofar Long Term Archive

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Conclusions Successful integration of Astro-WISE and EGEE Storage and processing integrated –Portable implementation  jLite  dCache SRM Power of Astro-WISE available for Lofar user community Future work on improving robustness Plan for demo at next user forum First Lofar observation of Cygnus A with three stations (Copyright: ASTRON)

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Web services Target processor

Enabling Grids for E-sciencE EGEE-III INFSO-RI Integration of Astro-WISE with Grid storage - EGEE Conference Barcelona Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd53‘) and (ReducedScienceFrame.filter == ‘#841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat (dependency) Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)