1. Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG 8 March 2007 Lofar Survey Workshop OmegaCEN NOVA – Kapteyn Institute -RuG 8 March 2007 Lofar Survey Workshop Edwin A. Valentijn

2. Lofar Survey workshop 8/3/2007 “Moore’s law” for N-body simulations Courtesy Simon White

3. Lofar Survey workshop 8/3/2007

4. Instrument: e-science = share resources beyond workstation Compute power: –AstroWise grid (RuG, Leiden, MPE, Bonn, INAF) –EGEE-3 Storage: – AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn, INAF) –BiG GRID (RuG- NikHef- SARA) Database: all meta + calibration +source data –AstroWise - Oracle Application programs + doc’s Web services Interoperability: EGEE- EURO-VO = share resources beyond workstation Compute power: –AstroWise grid (RuG, Leiden, MPE, Bonn, INAF) –EGEE-3 Storage: – AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn, INAF) –BiG GRID (RuG- NikHef- SARA) Database: all meta + calibration +source data –AstroWise - Oracle Application programs + doc’s Web services Interoperability: EGEE- EURO-VO

5. Lofar Survey workshop 8/3/2007 SDSS ROSAT 2MASS FIRST GAIA Scattered Archives

6. Lofar Survey workshop 8/3/2007 ROSAT FIRST GAIA SDSS2MASS Euro-VO

7. Lofar Survey workshop 8/3/2007  Manipulation  X-match  Visualization  Direct links to: CDS Aladin ALADIN

8. Lofar Survey workshop 8/3/2007 TOPCAT VOTable view

9. Lofar Survey workshop 8/3/2007 Multi-tier models for LHC & LOFAR Large Hadron Collider Tier-0 (CERN): –copy of all raw data Tier-1 (~8-10 centers) –archive 1/n fraction of raw data & reconstructed data –regular re-processing of the raw data –archiving data from Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –data analyses –no data archiving Large Hadron Collider Tier-0 (CERN): –copy of all raw data Tier-1 (~8-10 centers) –archive 1/n fraction of raw data & reconstructed data –regular re-processing of the raw data –archiving data from Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –data analyses –no data archiving LOFAR Tier-0 (RUG): central processor –lots of processing, limited archive Tier-1 (~6-8 centers) –specialized science analysis –advanced processing of (large) data sets –dedicated archive, also for Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –individual scientists –data analyses –no data archiving

10. Lofar Survey workshop 8/3/2007 AstroWise paradigm “Classical” paradigm Forward chaining Target processing - Awe Backward chaining waterfall model TIER architecture User hunts upstream driven by input raw data Driven by query of user Process in pipeline workflow Process in bits and pieces on the fly Backward chaining Operators push data User pulls data Results in releases Provide information system Static archives – publish Dynamic archives –publish Internet Raw data - obsolete Raw data is sacred

11. Lofar Survey workshop 8/3/2007 Virtual Survey Telescope NL:Leiden, Nijmegen, Groningen EU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago NL:Leiden, Nijmegen, Groningen EU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago raw pixel data  pipelines/cal files  catalogues all integrated in one information system 100% data lineage 1 dec 2006 delivered ! distributed services processing GRID Storage GRID Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system 100% data lineage 1 dec 2006 delivered ! distributed services processing GRID Storage GRID Methods/services GRID

12. Lofar Survey workshop 8/3/2007 Current Instruments/projects VST- OmegaCAM –GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans VISTA –UltraVISTA, VIKING via Edinb. SuprimeCAM @ Subaru WFI@2.2m, WFC@INGWFI@2.2m ACS@HST Coma legacy survey Panstarrs Lofar –Surveys –EoR VST- OmegaCAM –GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans VISTA –UltraVISTA, VIKING via Edinb. SuprimeCAM @ Subaru WFI@2.2m, WFC@INGWFI@2.2m ACS@HST Coma legacy survey Panstarrs Lofar –Surveys –EoR

13. Lofar Survey workshop 8/3/2007 Lofar Storage

14. Lofar Survey workshop 8/3/2007 Astro-Wise VO Properties Benefits integrated dynamic db on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID SETI@home Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

15. Lofar Survey workshop 8/3/2007 TARGET diagram Lofar Survey/ ST ACS

16. Lofar Survey workshop 8/3/2007 www.astro-wise.org

17. Lofar Survey workshop 8/3/2007 www.ASTRO-WISE/portal

18. Lofar Survey workshop 8/3/2007 Web services- object viewer

19. Lofar Survey workshop 8/3/2007 Web services- object maker compute GRID

20. Lofar Survey workshop 8/3/2007

21. Planning 2007 ADD with Astron - enable communities Publish WENSS in VO connect to EGEE AstroWise at Astron Build e-SOC at RC RuG Source extraction -> source parameters services Involve Lofar community ADD with Astron - enable communities Publish WENSS in VO connect to EGEE AstroWise at Astron Build e-SOC at RC RuG Source extraction -> source parameters services Involve Lofar community

22. Lofar Survey workshop 8/3/2007 e-science  me-science