Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy Unit, Institute for Astronomy, University of Edinburgh, UK

WFAU Science Archives 30th March 2015MW Astrophysics from WF Surveys2 EARLIER PROCESSING AT CASU WFAU

Roles of the Science Archive Repository of (e.g. VISTA) data from reduced images to complex, catalogue products. Interface for survey teams and community to explore survey products to do science. Ability to work with data from project and other data Interface for survey teams to check data for quality control. Requires both: a dynamic main-DB which is updated with new data, better calibration, reprocessing, quality control, higher order products. Static, well documented release-DBs that can be referred to in publications. 30th March 2015MW Astrophysics from WF Surveys3

Common Design WSA, VSA, OSA developed through VDFS – VISTA Data Flow System Similar input image and catalogue FITS files generated by CASU MEF format, header info, table columns etc Common software architecture, data pipeline, user interfaces, hardware architecture. Instrument dependent functionality (as well as programme) GES is significantly different: spectroscopic, wider range of data from multiple groups, different interfaces BUT uses a lot of the underlying philosophy, software architecture, hardware designs. 30th March 2015MW Astrophysics from WF Surveys4

VST- ATLAS Gaia-ESO Spectroscopic VISTA VVV, VHS, [VMC]UKIDSS GCS, GPS, LAS; UHS Milky Way Surveys 30th March 2015MW Astrophysics from WF Surveys5

Archives: 30th March 2015MW Astrophysics from WF Surveys6

Imaging Surveys Pipeline Ingest nightly processed image and catalogue data from CASU Provenance - link related images Quality Control: Automated + input from teams. Link tile and pawprint data (VISTA) Process data for semester - done per programme : Produce and ingest deep stacks/tiles/mosaics + catalogues Merge pass-band catalogues to create source tables Create neighbour tables to link external catalogues Link multi-epoch data and calculate variability statistics Release a documented, static data product to users Create useful interface tools for users to query specific data, view and analyse it Convert data to ESO format and copy to ESO (VISTA, VST) 30th March 2015MW Astrophysics from WF Surveys7

GES Archive Spectroscopic survey of ~100,000 stars, covering all major components of MW using VLT-FLAMES Archive includes: list of targets with photometry used in the target selection spectra processed by CASU atomic and molecular line lists Recommended astrophysical parameters derived by various working groups: T eff, log(g), abundances … Neighbour tables to other surveys: SuperCOSMOS, VST- ATLAS, 2MASS, VISTA-VHS No public release yet. 3 rd internal data release. Public releases in a few months time. 30th March 2015MW Astrophysics from WF Surveys8

Value Added Data Data created by science teams: VMC : PSF catalogues Variable types (Cepheid, Eclipsing Binary) VVV: PSF catalogues (1 epoch) [Not yet delivered] Proper motion [Not yet delivered] GES: Wide range of parameters from multiple working groups. Work to improve connections of team generated data with CASU/WFAU generated to maximise usefulness. 30th March 2015MW Astrophysics from WF Surveys9

3D Extinction Maps Included VVV/Glimpse extinction maps of bulge (Chen et al. 2013) Join to vvvSource table. Filter conversions for VISTA, UKIRT, VST, Spitzer and WISE filters Easily produce extinction corrected SEDs, CMDs. Would like to add in more (IPHAS, GES …) 30th March 2015MW Astrophysics from WF Surveys10

Survey Interfaces (see DEMO) Coverage Maps Archive Listing GetImage / Multi-GetImage Region CrossID Freeform SQL Enhanced version ADQL (OSA only so far) 30th March 2015MW Astrophysics from WF Surveys11

Data Volumes Images: WFCAM 1720 raw image frames a day (4 2kx2k) VISTA 580 raw image frames a day (16 2kx2k) VST 100 raw image frames a day (32 4kx2k) ~1PB of image data Catalogues: VVV completely dominates: combination of high source density, short exposure time. Next VVV release will be 40TB 55 billion detections. Largest astronomical SQL database! 30th March 2015MW Astrophysics from WF Surveys12

Surveys are not independent entities Importance of multi-wavelength astronomy Wide range in temperatures, dust extinction Rare objects (e.g. high-z QSOs, BDs, odd transients) Common survey fields VST-ATLAS and VHS UKIDSS LAS/UHS and SDSS VVV, GPS, IPHAS, VPHAS+ Need for data integration Cross-neighbours tables, publishing to VO Matched aperture photometry 30th March 2015MW Astrophysics from WF Surveys13

WFAU VDFS Publications Hambly et al. 2008, MNRAS, 384, 637 (WSA) Cross et al. 2009, MNRAS, 399, 1730 (Multi-Epoch processing) Cross et al. 2012, A&A, 548A, 119 (VSA) Various other technical documents: 30th March 2015MW Astrophysics from WF Surveys14

Future work Matched Aperture Pipeline: Basic pipeline complete, BUT bugs and many additions necessary Between surveys, inter survey, user run Variables – see talk by Carlos Ferreira-Lopes: Add in improved indices for variable selection Add in period-finding Proper Motions. Flagging of bright stars. Interface improvements Cross-programme selection (for public available data) Better connections to Value-Added products. 30th March 2015MW Astrophysics from WF Surveys15

Please speak to me … Current users: What could be improved? What is missing? PIs of new VISTA/VST surveys or small programmes: If you want the pipeline outputs / interfaces Producers of VA products (e.g. 3D extinction maps) We can host them linked to useful data PIs of new imaging / spectroscopic surveys who need data management. We have the experience 30th March 2015MW Astrophysics from WF Surveys16