The LSST Data Processing Software Stack Tim Jenness (LSST Tucson) for the LSST Data Management Team Abstract The Large Synoptic Survey Telescope (LSST)

Slides:



Advertisements
Similar presentations
Configuration management
Advertisements

Configuration management
(and especially Robert Lupton and Dustin Lang)
Chapter 4 Quality Assurance in Context
HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.
Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
11 Games and Content Session 4.1. Session Overview  Show how games are made up of program code and content  Find out about the content management system.
Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)
Framework for Automated Builds Natalia Ratnikova CHEP’03.
Data Management Subsystem: Data Processing, Calibration and Archive Systems for JWST with implications for HST Gretchen Greene & Perry Greenfield.
DCS Overview MCS/DCS Technical Interchange Meeting August, 2000.
National Center for Supercomputing Applications Observational Astronomy NCSA projects radio astronomy: CARMA & SKA optical astronomy: DES & LSST access:
1 Radio Astronomy in the LSST Era – NRAO, Charlottesville, VA – May 6-8 th LSST Survey Data Products Mario Juric LSST Data Management Project Scientist.
IXYZ Frank Marshall NASA/GSFC 25 April 2012 April 25, 20121Implementing Portals of the Universe.
The Pipeline Processing Framework LSST Applications Meeting IPAC Feb. 19, 2008 Raymond Plante National Center for Supercomputing Applications.
Dec 2, 2014 MAST Data Discovery Portal Tom Donaldson Tony Rogers.
Magnetic Field Measurement System as Part of a Software Family Jerzy M. Nogiec Joe DiMarco Fermilab.
Principles of Information Systems, Sixth Edition Systems Design, Implementation, Maintenance, and Review Chapter 13.
Configuration Management (CM)
Prototype system of the Japanese Virtual Observatory The Japanese Virtual Observatory (JVO) aims at providing easy access to federated astronomical databases.
Object-Oriented Modeling Chapter 10 CSCI CSCI 1302 – Object-Oriented Modeling2 Outline The Software Development Process Discovering Relationships.
OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative OOI Cyberinfrastructure Architecture Overview Michael Meisinger Life Cycle Architecture Review.
NOAO Brown Bag Tucson, AZ March 11, 2008 Jeff Kantor LSST Corporation Requirements Flowdown with LSST SysML and UML Models.
LSST: Preparing for the Data Avalanche through Partitioning, Parallelization, and Provenance Kirk Borne (Perot Systems Corporation / NASA GSFC and George.
Introduction to Making Multimedia
DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Goals and Objectives Jeff Kantor DM System Manager Tim Axelrod DM System Scientist.
Software Engineering Prof. Ing. Ivo Vondrak, CSc. Dept. of Computer Science Technical University of Ostrava
Pachon Sky Camera Christopher Stubbs Chuck Claver Jan 25, 2014.
EScience May 2007 From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith NOAO/CTIO, LSST.
Quality Assurance Benchmark Datasets and Processing David Nidever SQuaRE Science Lead.
Data Analysis Software Development Hisanori Furusawa ADC, NAOJ For HSC analysis software team 1.
The Active Optics System S. Thomas and the AO team.
1 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Name of Meeting Location Date - Change in Slide Master Data Release Processing.
Counting individual galaxies from deep mid-IR Spitzer surveys Giulia Rodighiero University of Padova Carlo Lari IRA Bologna Francesca Pozzi University.
Conversion of Stackfit to LSST software stack Status as of Feb 20, 2012.
Configuration Management and Change Control Change is inevitable! So it has to be planned for and managed.
What the Data Products Program Offers Users Todd Boroson Dick Shaw Presentation to NOAO Users Committee October 23, 2003.
Principles of Information Systems, Sixth Edition 1 Systems Design, Implementation, Maintenance, and Review Chapter 13.
COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 12 Science Data Processing
The Large Synoptic Survey Telescope Project Bob Mann LSST:UK Project Leader Wide-Field Astronomy Unit, Edinburgh.
 Programming - the process of creating computer programs.
Ray Plante for the DES Collaboration BIRP Meeting August 12, 2004 Tucson Fermilab, U Illinois, U Chicago, LBNL, CTIO/NOAO DES Data Management Ray Plante.
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
Mountaintop Software for the Dark Energy Camera Jon Thaler 1, T. Abbott 2, I. Karliner 1, T. Qian 1, K. Honscheid 3, W. Merritt 4, L. Buckley-Geer 4 1.
What is Firefly (1) A web UI framework for web applications
Implementation Review1 Archive Ingest Redesign March 14, 2003.
Current Status of Users Meeting F URUSHO, Reiko (Astronomy Data Center, NAOJ) Topics: 1. Minor Bodies Search is Opened. 2. Number.
From Missions to Measurements: an Ocean Discipline Experience.
26th October 2005 HST Calibration Workshop 1 The New GSC-II and it’s Use for HST Brian McLean Archive Sciences Branch.
PROGRAMMING FUNDAMENTALS INTRODUCTION TO PROGRAMMING. Computer Programming Concepts. Flowchart. Structured Programming Design. Implementation Documentation.
Distributed parallel processing analysis framework for Belle II and Hyper Suprime-Cam MINEO Sogo (Univ. Tokyo), ITOH Ryosuke, KATAYAMA Nobu (KEK), LEE.
GSPC -II Program GOAL: extend GSPC-I photometry to B = V ˜ 20 add R band to calibrate red second-epoch surveys HOW: take B,V,R CCD exposures centered at.
LECTURE 5 Nangwonvuma M/ Byansi D. Components, interfaces and integration Infrastructure, Middleware and Platforms Techniques – Data warehouses, extending.
1 LSST DESC MEETING | SLAC NATIONAL LABORATORY | MARCH 8, 2016 Name of Meeting Location Date - Change in Slide Master LSST Data Management Brief Status.
Wednesday NI Vision Sessions
1 GMT Community Science Meeting Monterey, CA October 1 – 3, 2015 LSST – A Discovery Machine for ELT Era Science Beth Willman LSST Deputy Director GMT Community.
1 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Investigating interoperability of the LSST Data Management.
The GLORIA demonstrator experiment Ariel Majcher National Centre for Nuclear Research Warsaw, Poland XXXII IEEE-SPIE Joint Symposium Wilga 2013, May 29.
Pan-STARRS observational requirements specification.
Pi of the Sky off-line experiment with GLORIA Ariel Majcher National Centre for Nuclear Research Warsaw, Poland 10th INTEGRAL/BART Workshop, April.
T. Axelrod, NASA Asteroid Grand Challenge, Houston, Oct 1, 2013 Improving NEO Discovery Efficiency With Citizen Science Tim Axelrod LSST EPO Scientist.
LSST Commissioning Overview and Data Plan Charles (Chuck) Claver Beth Willman LSST System Scientist LSST Deputy Director SAC Meeting.
LSST Commissioning Overview and Data Plan Charles (Chuck) Claver Beth Willman LSST System Scientist LSST Deputy Director SAC Meeting.
Overview – SOE PatchTT November 2015.
Optical Survey Astronomy DATA at NCSA
Single Object & Time Series Spectroscopy with JWST NIRCam
IEEE Std 1074: Standard for Software Lifecycle
Chapter 1 Introduction(1.1)
SCEC-VDO vtk Summer 2018 Objectives.
Presentation transcript:

The LSST Data Processing Software Stack Tim Jenness (LSST Tucson) for the LSST Data Management Team Abstract The Large Synoptic Survey Telescope (LSST) is an 8-m optical ground-based telescope being constructed on Cerro Pachón in Chile. LSST will survey half the sky every few nights in six optical bands. The data will be transferred to NCSA and within 60 seconds they will be reduced using difference imaging techniques and detected transients will be announced to the community in the VOEvent format. Annual data releases will be made from all the data during the 10-year mission, with unprecedented depth of coadds and time resolution of catalogs for such a large region of sky. Here we present the current status of the data processing software, and describe how to obtain it. More Information Summer 2015 Release Article: arXiv: LSST Design Overview: arXiv: LSST Science Book: arXiv: LSST Data Products: DM Applications Design: Key Numbers: The LSST Software Stack The Large Synoptic Survey Telescope will take about 15 TB of image data per night and after ten years of operations will have 15 petabytes of catalog data from the final data release, and 0.5 exabytes of image data. We are writing a suite of software packages to enable these data products to be created with sufficient quality and performance to meet the established science goals. The science pipeline software enables two key components of the data management system. The Alert Production pipelines (also known as Level 1) process the data from the telescope and publish alerts to the community within 60 seconds of data acquisition. Data Release Production (Level 2) is responsible for the annual data releases which reprocess all the data each year to generate the best possible catalogs. Both these systems are integrated with the Calibration Products Production that continuously calculates the best calibrations for the pipelines. The software also provides a toolkit for user-supplied code that can be used to efficiently and effectively analyze LSST data as part of Level 3 processing or their own Pipelines. Stack Components daf The Data Access Framework is responsible for mediating between the archive resources and the application writer. The pipeline code has a completely abstract view of file I/O and only has to know how to deal with data objects representing fundamental types such as exposures and tables. Currently FITS is the internal format but the system is designed such that the internal format could be changed to HDF5, for example, and no changes would have to be made to the science pipeline code. This abstraction of the files from the code protects us against shifts in format preferences. afw The Astronomy FrameWork provides the core classes for manipulating exposures and catalogs, including detecting sources and world coordinate handling. ip These are the image processing classes, including packages for instrument signature removal and image differencing. meas The measurement packages include code for determining source properties and correcting astrometry and photometry. obs These classes provide instrument-specific knowledge to the software system, providing information to the data access framework to teach it how to interpret data from a range of optical cameras. The obs packages currently support data from some instruments on Subaru and CFHT, in addition to simulated LSST data. Work is ongoing to add support for DECam. pipe Pipeline infrastructure and tasks. A task is the name for a core processing component that can be chained with other tasks to build a pipeline. Data flow for annual Data Release Production. Note how Level 1 products are re- calculated for each release. Obtaining the Software Source code: Summer 2015 release: Known to work on CentOS 6 and 7 & OS X Yosemite & Mavericks. GCC or later, or Apple clang 6, required. We are experimenting with a CernVM-FS binary distribution for Linux and OS X. Summer 2015 Release Whilst the software is open source and can be installed at any time, LSST makes formal releases of the science pipeline software at the end of each six month development cycle in the spring and autumn. The most recent release covered the summer development cycle and was labeled Summer 2015 and released in September Detailed release notes can be found at here we provide a summary. Multi-band processing for coadds: New command-line tasks have been added for consistent multi-band processing of coadds. This new data processing flow carefully combines source measurements taken in multiple bands to guarantee consistent deblending across all bands, including when carrying out forced photometry, thereby enabling reliable color measurement, and ensuring that all sources are measured in each band, regardless of the bands where they are detected. Upgraded astrometry calculation: Previously astrometry was calculated using astrometry.net code and related catalogs distributed by LSST. To improve flexibility in the code the astrometry fitter is now pluggable and now includes an alternative implementation. Support for PSFex: PSFEx is currently the state of the art external package for point spread function (PSF) determination, used in projects such as DES. LSST wrappers were created such that PSFEx could be used as a plugin in place of the built in PSF determiner. Aperture correction: Aperture corrections are now applied consistently throughout the stack. More efficient handling of large footprints: A footprint defines the pixels associated with a particular source or blended sources. This release saw significant improvements in performance when using very large footprints. Enable use of deblended heavy footprints in coadd forced photometry: Given the new multi-band processing for coadds we now have a reference catalog that is consistent across all bands. This catalog allows the use of the source's heavy footprints (A heavy footprint is a footprint that includes the pixel values) to replace neighbors with noise in forced photometry, thus providing deblended forced photometry and consistent deblending across all bands. This provides much better colors for blended objects as well as measurements for drop-out objects that do not get detected in the canonical band. This functionality has been enabled for forced coadd photometry. Significant improvements in the table class: The AFW package has a native C++ implementation of a class for manipulating table data for handling the results of detections and measurements algorithms. This release comes with some major enhancements to the internals of afw.table and, in particular, much better support for compound fields (such as Right Ascension/Declination tuples). Device independent displays: DS9 is no longer hard-wired into the software and the choice of display tool is now user configurable. The intention is for the next release to include support for the Firefly visualization tool. Stack Summary About 40 distinct packages. 100,000 lines of Python, 110,000 lines of C++. Migrating to C++11 and support of Python 3. Left: Three-colour (R, I, Y) composite image of an 11 by 11 arcminute patch of SDSS Stripe 82 based on Hyper Suprime-Cam engineering test data processed through the LSST stack Summer 2015 release.