1. EURO-VO workshop 1 July 2005 A Virtual Survey SYSTEM Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006 Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006 Edwin A. Valentijn

2. EURO-VO workshop 1 July 2005 Basic objectives Wide Field imaging EU Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs) Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)  (O)MegaCAM

3. EURO-VO workshop 1 July 2005 status VO conference 2002 - design Build information system - working Implemented, qualified WFI@2.2m, WFC@INT, MDM, OmegaCAM@ILT >>Qualify with OmegaCAM@VST-2006 >>tune to run Public Surveys >>quality control >>Optimize federation/replication VO conference 2002 - design Build information system - working Implemented, qualified WFI@2.2m, WFC@INT, MDM, OmegaCAM@ILT >>Qualify with OmegaCAM@VST-2006 >>tune to run Public Surveys >>quality control >>Optimize federation/replication

4. EURO-VO workshop 1 July 2005 new paradigm target processing from: -waterfall/ multi-tier -data pushing -raw data processing -raw data delete -result ->archive -”releases” from: -waterfall/ multi-tier -data pushing -raw data processing -raw data delete -result ->archive -”releases” to: -hunting /full linking -data pulling -target processing -raw data -archive -all in archive -request driven to: -hunting /full linking -data pulling -target processing -raw data -archive -all in archive -request driven raw pixel data  pipelines/cal files  catalogues all integrated in one information system distributed services  Virtual Survey Telescope processing GRID Storage GRID Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system distributed services  Virtual Survey Telescope processing GRID Storage GRID Methods/services GRID

5. EURO-VO workshop 1 July 2005 Astro-Wise VO Properties Benefits integrated dynamic db on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

6. EURO-VO workshop 1 July 2005 components Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joints Fileserver – distributed- via db Python clients CVS distributed code base - opipe Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joints Fileserver – distributed- via db Python clients CVS distributed code base - opipe

7. EURO-VO workshop 1 July 2005 Astro-Wise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

8. EURO-VO workshop 1 July 2005 Target processing: ++ the make metaphor awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcesTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcesTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive

9. EURO-VO workshop 1 July 2005 Intra-operability peer to peer code base + docs : CVS Db: “Advanced Replication” evolving to streaming code base + docs : CVS Db: “Advanced Replication” evolving to streaming WRITE –READ-ONLY WRITE –REPLICATION

10. EURO-VO workshop 1 July 2005 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

11. EURO-VO workshop 1 July 2005 Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y) # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)

12. EURO-VO workshop 1 July 2005 Astro-Wise PORTAL

13. EURO-VO workshop 1 July 2005 Web services- object viewer

14. EURO-VO workshop 1 July 2005 QC - calibration scientist monitoring

15. EURO-VO workshop 1 July 2005 QC - calibration scientist monitoring

16. EURO-VO workshop 1 July 2005 Web services- object maker

17. EURO-VO workshop 1 July 2005 VST - Virtual Survey Telescope

18. EURO-VO workshop 1 July 2005 Lofar IBM- Blue Gene/L

19. EURO-VO workshop 1 July 2005 Thanks! Welcome at next Astro-Wise tutorial! October 2005 Netherlands www.astro-wise.org Welcome at next Astro-Wise tutorial! October 2005 Netherlands www.astro-wise.org