1. Gijs Verdoes Kleijn Edwin Valentijn Marjolein Cuppen for the Astro-WISE consortium

2.  What is Astro-WISE?: An information accumulator for eScience  Massive datasets  Data centric  Federated  Paradigm for Citizen eScience? Astronomers and public collaborating in one system

3.  Goals Where it started: data avalanches from Wide-field imaging surveys: OmegaCAM/VST  Needles in haystacks  Haystacks Share: federate distributed resources (hardware + science teams) All-in-one information system: publishing, archiving, calibration, post- calibration analysis  Concepts: the paradigm Data model in object model Extreme backwards-chaining All All I/O through database: VirtualTelescopePlaat met vereenvoudiging  Implementation Python as glue  Try yourself:

4. UKIDSS workshop 2007 VIRCAM@VISTA 0.6 sq.deg. NIR camera 16 2kx2k detectors 0.35” pixels Science verification phase OmegaCAM @VST 1sq.deg. Opt. camera 32 2kx4k detectors 0.21” pixels Commissioning mid 2010 ESO Public Imaging Surveys VIKING~250nights z, Y, J, H, K KIDS~500 nights 1500 sq.deg. u g r i

5. –OmegaCAM 24% (~2500 GB) –VIRCAM 72% –(~7500 GB) –All Current Paranal –Instruments 4% (433.2 GB) Paranal Monthly Data Rates 2007 statistics –Courtesy Mark Neeser

6. Science drivers  Needles in haystacks high-z QSOs (6.4<z<7.2) Extreme Galactic White Dwarfs (Rare) AGNs The unanticipated  Haystacks: Dark matter distribution: weak lensing Dark energy Evolution of galaxy clusters Unexpected  Needles are easier than haystacks

7.  Facilitate (un)foreseen uses of calibrated and raw Tera/Petabyte data sets by global pools of collaborators and resources  This requires enabling changes at any time in: 1. Physical parameters (e.g., instrument/atmosphere) 2. Improved algorithms 3. Improved insight in 1. or 2.  Requires a single integrated information system Survey monitoring Quality control User tuned research Archiving (raw+final) Dissemminating results Calibration Integrated Information system

8.  Pool via federation into a single system Distributed science teams Distributed storage, compute and database resources  Peer-to-peer network

9. TARGET diagram Data model in object model OOP (python), inheritance, attribute persistency Full backwards-chaining of dependencies The philosophy that drives how we store information

10. Extreme data lineage

11. Astro-WISE information system data centric architecture  All data beyond pixel data is Metadata all pixel data data servers all Metadata database  all I/O to database  Compute clusters / GRIDs  Graphical User Interfaces (Web- services)  All components scalable

12. OmegaCEN review 2008 Astro-WISE - Virtual Survey Telescope CPUs DATA SERVERS: Images: Calibration & science Raw & final DATABASE LINKS METADATA images CATALOGS Contains “ALL input/output” Pipeline Methods/Algorithms In on-line repository Python (wrapped)

14. XLDB -Lyon Aug 2009 www.astro-wise.org

15. + MegaCAM, LBC, ISAAC, LOFAR, SDSS DR7, 2MASS, USNO…

16. Web services Target processor

17. Quality view

18.  Both final and raw products accessible  GUI can be fully web services no local installation needed to use it  can be installed everywhere with off-the-shelf hardware  Objects are linked, all bits are traced: on-the fly re-processing Annotation record building Built–in workflow  Easy publishing via Virtual Observatory and webservices  Own compute GRID operational  A federated integrated-system: Naturally serves global collaborative projects Scalable: image server/database/compute GRID resources can grow via federation  Mydb environment operational for individual research  Quick scripting: many tasks are “5 Line Scripts”  Fully user tunable – own provided algorithms/scripts allowed  Python: existing software can be wrapped (implemented for Sextractor, swarp, being implemented for GALPHOT, GALFIT) ( )

19.  Discovery of the exotic: gravitational lenses, mergers,… SNe, transients, Satellites asteroids  Survey Quality Control -visual inspection KIDS: 3yr manpower. -object masking -astrometry

20.  Open-up massive datasets to the public Playing -> exploring -> science A single environment for public and scientist  Feeling at home in the Universe on spaceship Earth

21.  Open-up massive datasets to the public Continum: playing -> exploring -> science A single environment for public and scientist  Feeling at home in the Universe on spaceship Earth www.astro-wise.org www.rug.nl/TarGet

23. TASK 24 www.rug.nl/ TarGet 2009-13 32M