1. Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray Plante (University of Illinois National Center for Supercomputing Applications), Kian-Tat Lim (Stanford Linear Accelerator Center), Jeffrey Bartels (Cirrus Enterprises )

2. ADASS September 23-26, 2007 London, UK The LSST Science Data Archive Millions of images in Image Archive Billions of astronomical objects in Object Catalog Trillions of sources in Source Catalog All will be stored and preserved for decades Data will be stored in file systems (images) and database management storage systems (all else) Access patterns will vary from very frequent small queries to scientific analyses of large % of the entire dataset

3. ADASS September 23-26, 2007 London, UK Data Acquisition Interface High- Speed Storage 1 Mountain Summit/Base Facility Key: Dashed Green: near- real-time data flow, validated in DC1, DC2 Dotted Red: nightly or longer data flow, validated in DC3 Solid Blue: on-demand data flow, validated in DC4 3 Data Access Centers 4 End User Sites Mixed- Speed Storage Data Access Servers Medium- Speed Storage Pipeline Servers 2 Archive Center Archive Ops Servers High- Speed Storage Camera Instrument Subsystem Observatory Control System Pipeline Servers DMS data flows

4. ADASS September 23-26, 2007 London, UK LSST Middleware

5. ADASS September 23-26, 2007 London, UK Data Access Framework Application Programming Interface (API) Data must be persisted in a variety of formats for development, testing, and operations –Composite C++ and python objects for pipeline processing –Relational database tables for persistence in the archive –FITS, VOTable, serialize object files for external access and tool integration Challenges in providing an API that meets requirements –Robust, transactional, high-speed –Encapsulate the underlying formatting and storage system architecture –Provide uniform handling for metadata and provenance API is a work in progress, first release in Data Challenge 2

6. ADASS September 23-26, 2007 London, UK LSST data and Middleware API

7. ADASS September 23-26, 2007 London, UK Persistence

8. ADASS September 23-26, 2007 London, UK Formatter

9. ADASS September 23-26, 2007 London, UK Storage

10. ADASS September 23-26, 2007 London, UK Status of work Software is open source and available to the public (but you should wait a bit…) The first release of the Data Access Framework is being done as part of Data Challenge 2 We are in system integration phase in September, 2007 We will execute with 2.5 TB of CFHTLS-Deep data and TALCS data in October - November, 2007 Results will be published in January, 2008