1 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Name of Meeting Location Date - Change in Slide Master LSST Alert Production Pipelines Simon Krughoff for the DM team LSST Data Management at UW Hot-Wiring the Transient Universe May 12-15, Santa Barbara

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Basic idea behind LSST: a uniform sky survey − 90% of time will be spent on a uniform survey: every 3-4 nights, the whole observable sky will be scanned twice per night − after 10 years, half of the sky will be imaged about 1000 times (in 6 bandpasses, ugrizy): a digital color movie of the sky − ~100 PB of data: about a billion 16 Mpix images, enabling measurements for 37 billion objects Left: a 10-year simulation of LSST survey: the number of visits in the r band (Aitoff projection of eq. coordinates) LSST in one sentence: An optical/near-IR survey of half the sky in ugrizy bands to r~27.5 (36 nJy) based on 1000 visits over a 10-year period: deep wide fast.

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Required system characteristics − Large primary mirror: (at least 6m) to go faint and to enable short exposures (30 s) − Agile telescope: 5 sec for slew and settle − Large field of view: enable fast surveying − Impeccable image quality: weak lensing − Fast Readout: ~2 sec for 3 GPix camera − Sophisticated software: 20 TB/night, 37 billion objects, 30 trillion measurements

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 LSST Telescope 8.4m, 6.7m effective 5 sec slew & settle

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 The field-of-view comparison: Gemini vs. LSST

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 LSST camera

HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 LSST camera Modular design: 3200 Megapix = 189 x16 Megapix CCD 9 CCDs share electronics: raft (=camera) Problematic rafts can be replaced relatively easily

8 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Petascale Computing, Gbps Networks The computing cluster at the LSST Archive at NCSA will run the processing pipelines. Single-user, single-application data center Commodity computing clusters. Distributed file system for scaling and hierarchical storage Local-attached, shared-nothing storage when high bandwidth needed Long Haul Networks to transport data from Chile to the U.S. 2x100 Gbps from Summit to La Serena (new fiber) 2x40 Gbps for La Serena to Champaign, IL (path diverse, existing fiber) Archive Site and U.S. Data Access Center NCSA, Champaign, IL Archive Site and U.S. Data Access Center NCSA, Champaign, IL Base Site and Chilean Data Access Center La Serena, Chile Base Site and Chilean Data Access Center La Serena, Chile

9 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 LSST From the User’s Perspective − A stream of ~1-10 million time-domain events per night, detected and transmitted to event distribution networks within 60 seconds of observation. − A catalog of orbits for ~6 million bodies in the Solar System. − A catalog of ~37 billion objects (20B galaxies, 17B stars), ~7 trillion observations (“sources”), and ~30 trillion measurements (“forced sources”), produced annually, accessible through online databases. − Deep co-added images. − Services and computing resources at the Data Access Centers to enable user-specified custom processing and analysis. − Software and APIs enabling development of analysis codes. Level 3 Level 1 Level 2

10 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Overview − Primary purpose: Satisfy science cases requiring rapid identification and follow-up (transients, fast-moving NEOs, etc.) − Transient science – Nova, supernova, GRBs – Source characterization – Instantaneous discovery − Nearby Solar System Objects – NEOs, PHAs

11 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Overview − Primary purpose: Satisfy science cases requiring rapid identification and follow-up (transients, fast-moving NEOs, etc.) − Transient science – Nova, supernova, GRBs – Source characterization – Instantaneous discovery − Nearby Solar System Objects – NEOs, PHAs

12 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Alert Production: Pipeline overview ( , , flux, …) DIASource Raw visit DiffimDetectMeasure ( , , flux, …) DIASource DIAObject SSObject Association Update DIAObject Alert Packet DIAObject record DIASource record DIASource records Construct Alert Transmit to event brokers LSST Simple broker Community Brokers

13 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Introducing Some Jargon − In normal survey mode, LSST will operate by capturing two back-to-back, 15- second exposures for each pointing. The two exposures are referred to as snaps They are combined to a visit, which is the basic input image product for Level 1 data products The primary purpose of the snaps is to enhance cosmic ray rejection − Level 1 processing will populate the Level 1 database tables, including: DIASource : table of sources detected on difference images DIAObject : objects and their characteristics, inferred from DIASource detection SSObject : the catalog of orbits of objects in the Solar System

14 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Pipeline: Source Detection − A visit is acquired and reduced (bias subtraction, flat fielding, cosmic ray rejection, combining of snaps etc.). − The visit image is then differenced against the appropriate template − DIASources are detected − DIASource position, flux, and shape are measured with multiple algorithms − The list is stored to a relational database for Level 1 data products (into the DIASource table) ( , , flux, …) DIASource Calibrated visit DiffimDetectMeasure

15 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Pipeline: Source Association − The Level 1 database is searched for a match with an existing DIAObject or an SSObject. If no match is found a new DIAObject is created If the DIASource has been associated with an SSObject, we issue an alert and stop. All further SSObject processing (eg., orbit recomputation) occurs in daytime. Otherwise, the associated DIAObject measurements are updated with new data (centroids, light-curves, etc.). − Forced photometry is performed for all DIAObjects overlapping the field of view to which a DIASource from this visit has not been associated. ( , , flux, …) DIASource DIAObject SSObject Association Update DIAObject

16 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Pipeline: Issuing Alerts − An alert packet is constructed that includes all previous DIASource measurements, all information contained in the DIAObject record, image cut-outs, and all metadata necessary to query the Level 1 database for a copy of this alert. − We plan to use VOEvent as the format for alert transmission − The packet will be transmitted to VOEvent Brokers We expect the community to provide sophisticated brokers with classification engines, cross-match capabilities to other catalogs, etc. LSST will provide a default, limited, broker Alert Packet DIAObject record DIASource record DIASource records Construct Alert Transmit to the internal or External Brokers

17 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Processing: System Architecture

18 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Processing: System Architecture

19 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Data Products − Processing to enable rapid detection and follow-up of time-domain events − Real-time image differencing as observing unfolds each night − Measurement of position, brightness and shape for each detection − Alerts to detected changes transmitted within 60 seconds of observing, enabling rapid follow-up Transient Detection with Image Differencing (CANDELS;

20 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1: Transients Alerts − LSST computing is sized for 10M alerts/night (average), 10k/visit (average), 40k/visit (peak) Dedicated networking for moving data from Chile to the US Dedicated image processing clusters New image differencing pipelines with improved algorithms − Will measure and transmit with each alert: position flux, size, and shape light curves in all bands (up to a ~year; stretch: all) variability characterization (eg., low-order light-curve moments, probability the object is variable) cut-outs centered on the object (template, difference image)

21 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 A note on the LSST broker # Keep only never-before-seen events within two # effective radii of a galaxy. This is for illustration # only; the exact methods/members/APIs may change. def filter(alert): if len(alert.sources) > 1: return False nn = alert.diaobject.nearest_neighbors[0] if not nn.flags.GALAXY: return False return nn.dist < 2. * nn.Re − No cross-match or classification − Intended for simple filtering − User defined filtering − See DPDD:

22 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Catalogs: DIASources − Includes measures of: Position Shape (adaptive Gaussian moments; Bernstein & Jarvis 2002) Model fits: -Point source model Measure of flux and position assuming the object is a stationary point source -Trailed source model Measure of flux, position, and direction of motion, assuming object moves sufficiently fast to trail in the image. Designed for Solar System objects. -Dipole model fit Fit the source with a “dipole” model, a positive next to a negative point source − Some DIASources will be false positives. We will flag suspect DIASources

23 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Catalogs: DIAObjects − Characterization of the underlying astrophysical objects detected in difference images Computed from associated DIASource records (recomputed as needed) Primary goal: Enable object classification − Include: Fits of position, parallax, and proper motion Mean point source flux (difference image and direct image) Variability characterization (e.g., Richards et al parameters) Pointers to nearby objects in the Level 2 catalog

24 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Level 1 Catalogs: SSObjects − Catalog of Solar System objects Asteroids, comets, KBOs, etc. − Includes: Orbital elements MOID Estimates of mean absolute magnitude (H, G) -Estimates performed in LSST bands − Convenience functions to compute the phase angle, reduced, H(  ), and absolute, H, asteroid magnitudes will be provided by the Level 1 database.

25 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Challenges − Difference imaging algorithms − Template generation − Flux dependent PSF − Many more… 0.6” FWHM Antilogus et al Becker et al. Winter 2014 Report

26 HOT-WIRING THE TRANSIENT UNIVERSE | SANTA BARBARA, CA | MAY 12-15, 201 Thanks!