1. Data provenance in astronomy Bob Mann Wide-Field Astronomy Unit University of Edinburgh (rgm@roe.ac.uk)

2. 2/24 Outline  Data and databases in astronomy  Case Study: UK Infrared Deep Sky Survey  Conclusions

3. 3/24 Outline  Data and databases in astronomy  Case Study: UK Infrared Deep Sky Survey  Conclusions

4. 4/24 Astronomers observe across the whole electromagnetic spectrum  Galaxy images look different across spectrum, due to:  Inherent angular resolution of the telescope  Different emission processes

5. 5/24 Astronomical data: original form  Different detector technologies used across the spectrum, yielding different types of data: e.g.  Ultraviolet/optical/infrared  Image: array of pixel values  X-ray  Event list: positions, arrival times, energies of all detected photons  Radio  Interferometric visibilities: sparse Fourier transform of a region of the sky

6. 6/24 Astronomical data: final form  Most research done using catalogue data  i.e. tables of attributes of detected sources – mainly discrete sources (stars, galaxies, etc)  Data compression  Catalogue – few % of image data volume  Amenable to representation in relational DB  Natural indexing by location in sky  …but original data products (images, spectra, event lists) sometimes needed

7. 7/24 Astronomical databases  Telescope archives  Heterogeneous collections of raw data files from all observations taken  Download data for reduction and analysis  Sky survey archives  Homogeneous data and pipeline reduction  “Science Archive” – do science on DB  Bibliographic archives – scans of journals

8. 8/24 Astronomical data processing  Data reduction  Remove instrumental signatures from raw data and produce “science-ready” data  Software packages written for specific instruments  Data analysis  Derive scientific results from science-ready data products – e.g. statistical analyses  Some astro-specific packages/environments – e.g. IRAF  Some use of programming languages  Fortran, C/C++, Python, Java  Some use of commercial packages  e.g. Interactive Data Language (IDL)

9. 9/24 Outline  Data and databases in astronomy  Case Study: UKIDSS  Introduction to UKIDSS  Data life-cycle in UKIDSS  Provenance in UKIDSS  Conclusions

10. 10/24 UK Infrared Deep Sky Survey  Set of five infrared sky surveys  Covering ~1/6 of the sky  From large/shallow to very small/very deep  See www.ukidss.org  Observations: 2005-2012 using Wide Field Camera (WFCAM) on UK Infrared Telescope (UKIRT) in Hawaii

11. 11/24 UKIDSS data life-cycle (1)  Summit of Mauna Kea  Data acquired from 4 WFCAM detectors  Summit pipeline: instrument health  Data written to LTO tape in NDF format  Tapes couriered to Cambridge weekly  Cambridge  Raw data converted from NDF to FITS  Data reduction pipeline run on nightly basis: ~100Gb/night  Remove instrumental signatures, combine images, detect and classify objects, calibrate positions & fluxes

12. 12/24 UKIDSS data life-cycle (2)  Edinburgh  Ingest data from Cambridge: catalogues into RDBMS; image metadata into RDBMS; images on disk  Combine data from multiple nights: generate new catalogues from stacked images  Prepare release databases for WFCAM Science Archive (WSA): see http://surveys.roe.ac.uk/wsa  Users worldwide  Extract raw images from Cambridge  Extract image and catalogues in FITS files from Edinburgh  Run queries on catalogues & image metadata in WSA

13. 13/24 Provenance in UKIDSS  Why is provenance important in UKIDSS?  What provenance information is recorded?  How will this be used?...and by whom?  …and is this adequate?

14. 14/24 Importance of provenance  Much UKIDSS science is rare object search Ratio of fluxes in H & K bands Ratio of fluxes in J & H bands Objects with these colours would be very unusual – and possibly very interesting. Are they real? Need ability to trace back to reduced image within which object was detected – maybe back to raw image.

15. 15/24 Structure of a FITS file Extensions Primary Header Primary Data Array Header Data Header Data Header: composed of 80-character ASCII records Data units can be images or tables

16. 16/24 FITS header records  Almost all records of the form KEYWORD = ‘ value ‘ / COMMENT  Some standard keywords defined, but considerable freedom to define new ones  Relevant metadata for particular instruments  Amongst standard set is HISTORY  Format: HISTORY free text  Provenance information can be stored in a series of HISTORY records

17. 17/24 UKIDSS FITS files (1)  Raw image files  Primary header: telescope/instrument set-up, observing conditions, target, observational parameters  Primary data array: empty  Extensions: (header,data) pairs for each of four detectors: header has detector-specific metadata; data is compressed image  Header keywords defined in Interface Control Document between Hawaii & Cambridge

18. 18/24 UKIDSS FITS files (2)  Reduced image files  Primary header & data array: metadata propagated from raw data file  Headers of extensions include HISTORY records for data reduction steps run at Cambridge, e.g HISTORY 20060615 17:30:02 HISTORY $Id: cir_stage1.c,v 1.11 2005/12/15 14:44:04 jim Exp $ HISTORY 20060615 17:31:04 HISTORY $Id: cir_qblkmed.c,v 1.9 2005/08/12 14:35:19 jim Exp $ HISTORY 20060615 17:32:36 HISTORY $Id: cir_xtalk.c,v 1.5 2005/10/17 14:58:50 jim Exp $ HISTORY 20060615 20:01:58 HISTORY $Id: cir_arith.c,v 1.8 2005/02/25 10:14:55 jim Exp $ What WhenWho

19. 19/24 UKIDSS FITS files (3)  Catalogue files  Primary header: metadata propagated from raw image  Primary data array: empty  Headers of extensions include metadata for catalogue generation process – invocations of software modules in HISTORY records, with parameter values in separate records  Header keywords for both reduced images and catalogues are defined in an Interface Control Document between Cambridge & Edinburgh

20. 20/24 User access to provenance info  All header records from all FITS files ingested into WSA except HISTORY records  So, users can track provenance through queries against WSA, and can get HISTORY records by downloading files  Hopefully enough to determined whether unusual object is real, but this is this good enough?

21. 21/24 Recap: Astronomical data processing  Data reduction  Remove instrumental signatures from raw data and produce “science-ready” data  Software packages written for specific instruments  Data analysis  Derive scientific results from science-ready data products – e.g. statistical analyses  Some astro-specific packages/environments – e.g. IRAF  Some use of programming languages  Fortran, C/C++, Python, Java  Some use of commercial packages  e.g. Interactive Data Language (IDL) ?

22. 22/24 Provenance in data analysis: Two main problems  Less controlled software environment  Little bits of code written for a specific analysis, not tried and tested pipeline modules  Use of data from many sources  UKIDSS/WSA is state-of-the-art for provenance  Many (esp. older) data resources not so good  Provenance of combined dataset only as good as provenance of worst constituent dataset?

23. 23/24 Does this matter?  Provenance information for data analysis is recorded in the journal paper (sort of)  Improving links between online literature and data sources  Increasing importance of large sky surveys with well controlled environments  Moving more of the data analysis from the user’s desktop to the data centre

24. 24/24 Conclusions  Modern sky survey systems record & publish extensive provenance for data reduction  Very little provenance recorded from data analysis – except description in journal paper  More could surely be done – but would researchers support overhead of doing so?  Improvements as more analysis in data centre  Could/should we be doing more?