Gijs Verdoes Kleijn Edwin Valentijn Marjolein Cuppen for the Astro-WISE consortium

 What is Astro-WISE?: An information accumulator for eScience  Massive datasets  Data centric  Federated  Paradigm for Citizen eScience? Astronomers and public collaborating in one system

 Goals Where it started: data avalanches from Wide-field imaging surveys: OmegaCAM/VST  Needles in haystacks  Haystacks Share: federate distributed resources (hardware + science teams) All-in-one information system: publishing, archiving, calibration, post- calibration analysis  Concepts: the paradigm Data model in object model Extreme backwards-chaining All All I/O through database: VirtualTelescopePlaat met vereenvoudiging  Implementation Python as glue  Try yourself:

UKIDSS workshop sq.deg. NIR camera 16 2kx2k detectors 0.35” pixels Science verification phase 1sq.deg. Opt. camera 32 2kx4k detectors 0.21” pixels Commissioning mid 2010 ESO Public Imaging Surveys VIKING~250nights z, Y, J, H, K KIDS~500 nights 1500 sq.deg. u g r i

–OmegaCAM 24% (~2500 GB) –VIRCAM 72% –(~7500 GB) –All Current Paranal –Instruments 4% (433.2 GB) Paranal Monthly Data Rates 2007 statistics –Courtesy Mark Neeser

Science drivers  Needles in haystacks high-z QSOs (6.4<z<7.2) Extreme Galactic White Dwarfs (Rare) AGNs The unanticipated  Haystacks: Dark matter distribution: weak lensing Dark energy Evolution of galaxy clusters Unexpected  Needles are easier than haystacks

 Facilitate (un)foreseen uses of calibrated and raw Tera/Petabyte data sets by global pools of collaborators and resources  This requires enabling changes at any time in: 1. Physical parameters (e.g., instrument/atmosphere) 2. Improved algorithms 3. Improved insight in 1. or 2.  Requires a single integrated information system Survey monitoring Quality control User tuned research Archiving (raw+final) Dissemminating results Calibration Integrated Information system

 Pool via federation into a single system Distributed science teams Distributed storage, compute and database resources  Peer-to-peer network

TARGET diagram Data model in object model OOP (python), inheritance, attribute persistency Full backwards-chaining of dependencies The philosophy that drives how we store information

Extreme data lineage

Astro-WISE information system data centric architecture  All data beyond pixel data is Metadata all pixel data data servers all Metadata database  all I/O to database  Compute clusters / GRIDs  Graphical User Interfaces (Web- services)  All components scalable

OmegaCEN review 2008 Astro-WISE - Virtual Survey Telescope CPUs DATA SERVERS: Images: Calibration & science Raw & final DATABASE LINKS METADATA images CATALOGS Contains “ALL input/output” Pipeline Methods/Algorithms In on-line repository Python (wrapped)

XLDB -Lyon Aug

+ MegaCAM, LBC, ISAAC, LOFAR, SDSS DR7, 2MASS, USNO…

Web services Target processor

Quality view

 Both final and raw products accessible  GUI can be fully web services no local installation needed to use it  can be installed everywhere with off-the-shelf hardware  Objects are linked, all bits are traced: on-the fly re-processing Annotation record building Built–in workflow  Easy publishing via Virtual Observatory and webservices  Own compute GRID operational  A federated integrated-system: Naturally serves global collaborative projects Scalable: image server/database/compute GRID resources can grow via federation  Mydb environment operational for individual research  Quick scripting: many tasks are “5 Line Scripts”  Fully user tunable – own provided algorithms/scripts allowed  Python: existing software can be wrapped (implemented for Sextractor, swarp, being implemented for GALPHOT, GALFIT) ( )

 Discovery of the exotic: gravitational lenses, mergers,… SNe, transients, Satellites asteroids  Survey Quality Control -visual inspection KIDS: 3yr manpower. -object masking -astrometry

 Open-up massive datasets to the public Playing -> exploring -> science A single environment for public and scientist  Feeling at home in the Universe on spaceship Earth

 Open-up massive datasets to the public Continum: playing -> exploring -> science A single environment for public and scientist  Feeling at home in the Universe on spaceship Earth

TASK 24 TarGet M