From photons to catalogs

Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy: I. Cluster Counts II. Weak Lensing III. Large-scale Structure IV. Supernovae Two multiband (photometric) surveys: 5000 deg 2 grizY to 24th mag AB griz deg 2 repeat (SNe) Build new 3 deg 2 FOV multi-CCD camera, Data management system, improve Blanco facilities Blanco 4-meter at CTIO 2/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla Credit:NOAO

11/08/2011APS-DPF 2011 DESDM I.Sevilla 3/29 SDSS-IIDES Area Nb. of CCDs2262 Resolution120 Mpix570 Mpix Raw data/night200 GB/night300 GB/night Catalog size18 TB100 TB (est.) Total data volume60 TB4 PB (est.) Credit: Kotwani et al. (2010)

Transfer Process Archive Distribute 4/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer CTIO  NCSA 300 GB/night in 18 h Process Archive Distribute 5/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer Process Orchestration: NCSA  HPC nodes Archive Distribute 6/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer Process Archive Results  Archive nodes Oracle DB Distribute 7/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer Process Archive Distribute Through web portals 8/29 11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Fermilab Team Huan Lin Svetlana Lebedeva Nelly Stanfield Douglas Tucker Munich Team Gurvan Bazin Art Carlson Joe Mohr (Project Scientist) SWG Development links Mike Jarvis + WL SWG (WL pipeline) John Marriner + SNe SWG (Diff Imaging) Molly Swanson + LSS SWG (Survey Masks) I.S. + LSS SWG (cosmic rays, satellites) SWG Testing links Clusters- Sarah Hansen, Wayne Barkhouse LSS- I.S. 11/08/2011APS-DPF 2011 DESDM I.Sevilla 9/29

Exposure consists of 62+ CCD images – 570 Mpix - 3deg 2 Survey is ~150, sec exposures over 525 nights Auxiliary CCDs record images for autoguiding and calibration 11/08/2011APS-DPF 2011 DESDM I.Sevilla 10/29 X calib. = RAW DATA we send this to NCSA

Correct for cross-talk among CCDs. Correct for bias levels, non-uniformities, other optical and electrical effects. 11/08/2011APS-DPF 2011 DESDM I.Sevilla 11/29 DETRENDED DATA Credit:NOAO

11/08/2011APS-DPF 2011 DESDM I.Sevilla 12/29 We need reference star catalogs Full focal plane is fit to single solution Correct optical distortion at focal plane Credit: E.Bertin We use Sextractor, SCAMP software by E.Bertin

REDUCED DATA 11/08/2011APS-DPF 2011 DESDM I.Sevilla 13/29 Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst): Make big least squares solution for a,b,k; apply results to science images. input output

At this point we have, for every night, approx. 300 exposures corrected by: Instrumental effects Absolute position Absolute photometry This is the nightly processing. We store these in the archive (+ auxiliary images, info). 11/08/2011APS-DPF 2011 DESDM I.Sevilla 14/29

11/08/2011APS-DPF 2011 DESDM I.Sevilla 15/29 COADDED DATA Go deeper; calibrate better BUT Point spread function is inhomogeneous: each exposure has different quality single exposure single exposure single exposure single exposure single exposure single exposure single exposure

11/08/2011APS-DPF 2011 DESDM I.Sevilla 16/29 COADDED DATA Find PSF in each image Homogeneize per coadd tile PSF HOMOGENEIZED 0.77 ’’ 1.32 ’’ 0.94 ’’

It takes 30x more time for coaddition with respect to nightly: PSF has to be extracted PSF has to be homogeneized Actual addition of image and recomputation of errors Additionally: Global photometric calibration among all images of the season We store these in the archive (+ auxiliary images, info). 11/08/2011APS-DPF 2011 DESDM I.Sevilla 17/29

11/08/2011APS-DPF 2011 DESDM I.Sevilla 18/29 CATALOGS E.Bertin This step is performed with the SExtractor package Position, shape and photometry is calculated.

11/08/2011APS-DPF 2011 DESDM I.Sevilla 19/29 Hundreds of millions of objects with hundreds of columns each:

11/08/2011APS-DPF 2011 DESDM I.Sevilla 20/29 Hundreds of millions of objects with hundreds of columns each:

11/08/2011APS-DPF 2011 DESDM I.Sevilla 21/29 Hundreds of millions of objects with hundreds of columns each:

11/08/2011APS-DPF 2011 DESDM I.Sevilla 22/29 Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. mag_band_g = 20.7 mag_band_r = 19.2 mag_band_i = 18.5 … + errors, other estimates (this only one kind of photo-z! More estimations foreseen) DESDM Photoz pipeline (neural network)

11/08/2011APS-DPF 2011 DESDM I.Sevilla 23/29 Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Eliminate instrumental signature Obtain true shape (intrinsic galaxy shape+ shear) DESDM WL pipeline local PSF

11/08/2011APS-DPF 2011 DESDM I.Sevilla 24/29 Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Credit: Pan-STARRS

11/08/2011APS-DPF 2011 DESDM I.Sevilla Produce cosmological simulations Process through atmosphere, detectors, include ‘nasty’ stuff 25/29

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 26/29 Galaxies Stars

11/08/2011APS-DPF 2011 DESDM I.Sevilla 27/29 BCS Images of First SPT Clusters Credit: Blanco Cosmology Survey Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic2 camera). Large scale management with SDSS data.

11/08/2011APS-DPF 2011 DESDM I.Sevilla The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products. Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of Raw and reduced images to be released yearly, catalogs at midpoint and end of survey. Community pipeline getting ready for usage of the DES camera starting May /29

11/08/2011APS-DPF 2011 DESDM I.Sevilla

11/08/2011APS-DPF 2011 DESDM I.Sevilla Development Funding ~$6M Development Funding ~$6M $4 million from NSF $4 million from NSF $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and Munich $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and Munich $300K from DES collaboration for Community Pipeline $300K from DES collaboration for Community Pipeline

(new simulations with more DES-like systematics coming up by Stanford team) Carnero et al. 2010

Perform full-depth observations in 100 sq.deg. Area is off main survey Overlap existing datasets when possible Run acceptance tests on data