Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz center Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Edwin A. Valentijn

Astro-Wise workshop Nov 2005 Goals 1.Tutorial to the system –New people 2.Prepare for projects and surveys –What to prepare? –Friday 3.Meet – collaboration- we share 1.Code, 2.Calibrations 3.Hardware 4.projects 1.Tutorial to the system –New people 2.Prepare for projects and surveys –What to prepare? –Friday 3.Meet – collaboration- we share 1.Code, 2.Calibrations 3.Hardware 4.projects

Astro-Wise workshop Nov 2005 Scope AstroWise provides infrastructure Course nov 2002 AstroWise  dec 2006 It is far, but not finished QC and photometry development will go on for many years AstroWise provides infrastructure Course nov 2002 AstroWise  dec 2006 It is far, but not finished QC and photometry development will go on for many years

Astro-Wise workshop Nov 2005 Basic objectives Wide Field imaging EU Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs) Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results 100’s Tbyte of image data and 10’s Tbyte of catalogue data With production spread over EU What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)  (O)MegaCAM

Astro-Wise workshop Nov 2005 status 2002: VO conference, course - design Build information system - working Implemented qualified MDM, >>Qualify with >>tune to run Public Surveys >>quality control >>Optimize federation/replication 2002: VO conference, course - design Build information system - working Implemented qualified MDM, >>Qualify with >>tune to run Public Surveys >>quality control >>Optimize federation/replication

Astro-Wise workshop Nov 2005

OmegaCAM All service observing We calibrate the instrument (not specific observtions) OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work First ILT version is available All service observing We calibrate the instrument (not specific observtions) OmegaCAM Consortium and AstroWise hosted projects share the calibrations, and the work First ILT version is available

Astro-Wise workshop Nov 2005 AstroWise paradigm “Classical” paradigm “hunting” AstroWise Target processing waterfall model TIER architecture User hunts upstream driven by input raw dataDriven by query of user Process in pipeline Process in bits and pieces on the fly Operators push dataUser pulls data Results in releasesProvide information system Static archives - publish Dynamic archives –publish on internet Raw data - obsoleteRaw data is sacred

Astro-Wise workshop Nov 2005 Paradigm raw pixel data  pipelines/cal files  catalogues all integrated in one information system Collected in database – Oracle 10g –Relational (table oriented db) but used for ++ code distributed services  Virtual Survey Telescope »processing GRID »Storage GRID »Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system Collected in database – Oracle 10g –Relational (table oriented db) but used for ++ code distributed services  Virtual Survey Telescope »processing GRID »Storage GRID »Methods/services GRID

Astro-Wise workshop Nov 2005 The avalanche integrated dynamic db on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing for everything 5LS: 5 Lines Script Awe> prompt Trend analysis Awe > prompt All dependent bits are traced “tell_me_everything_tool Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

Astro-Wise workshop Nov 2005 Components

Astro-Wise workshop Nov 2005 components Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joins Fileserver – distributed- via db Python clients CVS distributed code base - opipe Procedures + Cal plan at telescope Data model -> object model ++ ->++db Central db ; server/clients –All I/O except images –Meta data –Source lists = catalogues + associate lists –Links = references = joins Fileserver – distributed- via db Python clients CVS distributed code base - opipe

Astro-Wise workshop Nov Components - procedures calibration plan integrated

Astro-Wise workshop Nov 2005

Monitoring Photometric Calibration

Astro-Wise workshop Nov Components Data Model Sanity checks Quality control Calibration procedures Calibration procedures Image pipeline Source pipeline

Astro-Wise workshop Nov 2005 Astro-Wise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

Astro-Wise workshop Nov 2005 Object model

Astro-Wise workshop Nov 2005 Components 3: Database persistency Native talks PL-SQL Talks Python All data items = persistent Classes Nearly all I/O Fileserver Native talks PL-SQL Talks Python All data items = persistent Classes Nearly all I/O Fileserver

Tbyte source lists brains make the associations Link -lists as fast as possible

Astro-Wise workshop Nov 2005 VST - Virtual Survey Telescope

Astro-Wise workshop Nov 2005 Intra-operability peer to peer code base + docs : CVS Db: “Advanced Replication” evolving to streaming code base + docs : CVS Db: “Advanced Replication” evolving to streaming WRITE –READ-ONLY WRITE –REPLICATION

Astro-Wise workshop Nov 2005 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Co added images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

Astro-Wise workshop Nov 2005 Do It Do It

Astro-Wise workshop Nov 2005 Target processing: ++ the make metaphor awe> targethot=HotPixelMap.get(date=' ', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcessTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive awe> targethot=HotPixelMap.get(date=' ', chip='A5382') The processing chain is ReadNoise <-- Bias <-- HotPixels  > class HotPixelMap(ProcessTarget):  > > def self.make()  > class ProcessTarget():  > > def get(date, chip) # if not exist/up-to-date then make()  > > def exist() # does the target exist?  > > def uptodate() # is each dependency up to date? Fully recursive

Astro-Wise workshop Nov 2005 Example 5LS # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y) # Find ScienceFrames for a ccd named ccd53 and filter Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’) # From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q] # get the rms of the used Masterflat Awe> y = [k.flat.imstat.stdev for k in q] # Make a plot Awe> pylab.scatter(x,y)

Astro-Wise workshop Nov 2005 Astro-Wise PORTAL

Astro-Wise workshop Nov 2005 Web services- object viewer

Astro-Wise workshop Nov 2005 ESO-LV

Astro-Wise workshop Nov 2005 QC - calibration scientist monitoring

Astro-Wise workshop Nov 2005 QC - calibration scientist monitoring

Astro-Wise workshop Nov 2005 Web services- object maker

Astro-Wise workshop Nov 2005 Programme Afternoon Gijs Verdoes Kleijn –Demos –Exercises Afternoon Gijs Verdoes Kleijn –Demos –Exercises