1. Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center 14-18 Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz center 14-18 Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Edwin A. Valentijn

2. Astro-Wise workshop Nov 2005 Goals 1.Tutorial to the system –New people 2.Prepare for projects and surveys –What to prepare? –Friday 3.Meet – collaboration- we share 1.Code, 2.Calibrations 3.Hardware 4.projects 1.Tutorial to the system –New people 2.Prepare for projects and surveys –What to prepare? –Friday 3.Meet – collaboration- we share 1.Code, 2.Calibrations 3.Hardware 4.projects

3. Astro-Wise workshop Nov 2005 Scope AstroWise provides infrastructure Course nov 2002 AstroWise  dec 2006 It is far, but not finished QC and photometry development will go on for many years AstroWise provides infrastructure Course nov 2002 AstroWise  dec 2006 It is far, but not finished QC and photometry development will go on for many years

4. Astro-Wise workshop Nov 2005 Basic objectives Wide Field imaging EU Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs) Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)  (O)MegaCAM

5. Astro-Wise workshop Nov 2005 status 2002: VO conference, course - design Build information system - working Implemented qualified WFI@2.2m, WFC@INT, MDM, OmegaCAM@ILT >>Qualify with OmegaCAM@VST-2006 >>tune to run Public Surveys >>quality control >>Optimize federation/replication 2002: VO conference, course - design Build information system - working Implemented qualified WFI@2.2m, WFC@INT, MDM, OmegaCAM@ILT >>Qualify with OmegaCAM@VST-2006 >>tune to run Public Surveys >>quality control >>Optimize federation/replication

6. Astro-Wise workshop Nov 2005

7. OmegaCAM All service observing We calibrate the instrument (not specific observtions) OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work First ILT version is available All service observing We calibrate the instrument (not specific observtions) OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work First ILT version is available

8. Astro-Wise workshop Nov 2005 AstroWise paradigm “Classical” paradigm “hunting” AstroWise Target processing waterfall model TIER architecture User hunts upstream driven by input raw dataDriven by query of user Process in pipeline Process in bits and pieces on the fly Operators push dataUser pulls data Results in releasesProvide information system Static archives - publish Dynamic archives –publish on internet Raw data - obsoleteRaw data is sacred

9. Astro-Wise workshop Nov 2005 Paradigm raw pixel data  pipelines/cal files  catalogues all integrated in one information system Collected in database – Oracle 10g –Relational (table oriented db) but used for ++ code distributed services  Virtual Survey Telescope »processing GRID »Storage GRID »Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system Collected in database – Oracle 10g –Relational (table oriented db) but used for ++ code distributed services  Virtual Survey Telescope »processing GRID »Storage GRID »Methods/services GRID

10. Astro-Wise workshop Nov 2005 The avalanche integrated dynamic db on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

11. Astro-Wise workshop Nov 2005 Components

12. Astro-Wise workshop Nov 2005 components Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joins Fileserver – distributed- via db Python clients CVS distributed code base - opipe Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joins Fileserver – distributed- via db Python clients CVS distributed code base - opipe

13. Astro-Wise workshop Nov 2005 1- Components - procedures calibration plan integrated

14. Astro-Wise workshop Nov 2005

16. Monitoring Photometric Calibration

17. Astro-Wise workshop Nov 2005 2 Components Data Model Sanity checks Quality control Calibration procedures Calibration procedures Image pipeline Source pipeline

18. Astro-Wise workshop Nov 2005 Astro-Wise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

19. Astro-Wise workshop Nov 2005 Object model

20. Astro-Wise workshop Nov 2005 Components 3: Database persistency Native talks PL-SQL Talks Python All data items = persistent Classes Nearly all I/O Fileserver Native talks PL-SQL Talks Python All data items = persistent Classes Nearly all I/O Fileserver

21. Tbyte source lists brains make the associations Link -lists as fast as possible

22. Astro-Wise workshop Nov 2005 VST - Virtual Survey Telescope

23. Astro-Wise workshop Nov 2005 Intra-operability peer to peer code base + docs : CVS Db: “Advanced Replication” evolving to streaming code base + docs : CVS Db: “Advanced Replication” evolving to streaming WRITE –READ-ONLY WRITE –REPLICATION

24. Astro-Wise workshop Nov 2005 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

25. Astro-Wise workshop Nov 2005 Do It Do It

26. Astro-Wise workshop Nov 2005 Target processing: ++ the make metaphor awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcessTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcessTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive

27. Astro-Wise workshop Nov 2005 Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y) # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)

28. Astro-Wise workshop Nov 2005 Astro-Wise PORTAL

29. Astro-Wise workshop Nov 2005 Web services- object viewer

30. Astro-Wise workshop Nov 2005 ESO-LV

31. Astro-Wise workshop Nov 2005 QC - calibration scientist monitoring

32. Astro-Wise workshop Nov 2005 QC - calibration scientist monitoring

33. Astro-Wise workshop Nov 2005 Web services- object maker

34. Astro-Wise workshop Nov 2005 Programme Afternoon Gijs Verdoes Kleijn –Demos –Exercises Afternoon Gijs Verdoes Kleijn –Demos –Exercises