EURO-VO workshop 1 July 2005 A Virtual Survey SYSTEM Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006 Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO 5 year programme -> dec 2006 Edwin A. Valentijn

EURO-VO workshop 1 July 2005 Basic objectives Wide Field imaging EU Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs) Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)  (O)MegaCAM

EURO-VO workshop 1 July 2005 status VO conference design Build information system - working Implemented, qualified MDM, >>Qualify with >>tune to run Public Surveys >>quality control >>Optimize federation/replication VO conference design Build information system - working Implemented, qualified MDM, >>Qualify with >>tune to run Public Surveys >>quality control >>Optimize federation/replication

EURO-VO workshop 1 July 2005 new paradigm target processing from: -waterfall/ multi-tier -data pushing -raw data processing -raw data delete -result ->archive -”releases” from: -waterfall/ multi-tier -data pushing -raw data processing -raw data delete -result ->archive -”releases” to: -hunting /full linking -data pulling -target processing -raw data -archive -all in archive -request driven to: -hunting /full linking -data pulling -target processing -raw data -archive -all in archive -request driven raw pixel data  pipelines/cal files  catalogues all integrated in one information system distributed services  Virtual Survey Telescope processing GRID Storage GRID Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system distributed services  Virtual Survey Telescope processing GRID Storage GRID Methods/services GRID

EURO-VO workshop 1 July 2005 Astro-Wise VO Properties Benefits integrated dynamic db on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

EURO-VO workshop 1 July 2005 components Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joints Fileserver – distributed- via db Python clients CVS distributed code base - opipe Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joints Fileserver – distributed- via db Python clients CVS distributed code base - opipe

EURO-VO workshop 1 July 2005 Astro-Wise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

EURO-VO workshop 1 July 2005 Target processing: ++ the make metaphor awe> targethot=HotPixelMap.get(date=' ', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcesTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive awe> targethot=HotPixelMap.get(date=' ', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcesTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive

EURO-VO workshop 1 July 2005 Intra-operability peer to peer code base + docs : CVS Db: “Advanced Replication” evolving to streaming code base + docs : CVS Db: “Advanced Replication” evolving to streaming WRITE –READ-ONLY WRITE –REPLICATION

EURO-VO workshop 1 July 2005 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

EURO-VO workshop 1 July 2005 Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y) # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)

EURO-VO workshop 1 July 2005 Astro-Wise PORTAL

EURO-VO workshop 1 July 2005 Web services- object viewer

EURO-VO workshop 1 July 2005 QC - calibration scientist monitoring

EURO-VO workshop 1 July 2005 QC - calibration scientist monitoring

EURO-VO workshop 1 July 2005 Web services- object maker

EURO-VO workshop 1 July 2005 VST - Virtual Survey Telescope

EURO-VO workshop 1 July 2005 Lofar IBM- Blue Gene/L

EURO-VO workshop 1 July 2005 Thanks! Welcome at next Astro-Wise tutorial! October 2005 Netherlands Welcome at next Astro-Wise tutorial! October 2005 Netherlands