1 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Investigating interoperability of the LSST Data Management software stack with Astropy Tim Jenness LSST DM Deputy System Architect 26 th June 2016

2 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Large Synoptic Survey Telescope − 8.4m telescope imaging the entire Southern sky ~ twice a week for 10 years. − 3.2 gigapixel camera, ugrizy filters. − ~15TB per night continuously transferred to NCSA. − Alert Production Pipeline – Transient alerts issued within 60 seconds of the shutter closing. – Average about 10 million alerts per night. − Data Release Production Pipeline – 11 data releases over 10 years with a final image collection size of 0.5 exabyte and final catalog size of 15PB. – All data re-reduced for each data release.

3 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom LSST Data Management Science Pipeline Software − Software development began in − Evolved over ten years during research and development phase. − Mix of C++ and Python – 105k Python, 125k C++ (as of May 2016) − SWIG bindings (Python APIs look like C++ APIs)

4 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Astropy − Astropy grew out of a mailing list discussion in − Becoming the shared core for Python in Astronomy. − Focused on usability (of APIs and installation). − Specialist software in Astropy Affiliated Packages. − Rapid uptake in the community. − Will set community expectations for new generations of Python users. Should LSST science pipelines adopt Astropy core classes and contribute to Astropy?

5 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Astropy Adoption by LSST LSST are changing their approaches in the following areas: − Packaging − Coordinates (space and time) − Tables − Quantities and units Astropy also motivating us to look at alternate scheme for C++ bindings.

6 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Packaging and build system − EUPS – Version management – Packaging − Scons build system − Investigating a setup.py option

7 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Coordinates − Astropy time and sky coordinates classes much more complete than those in LSST software. – Based on ERFA/SOFA − LSST C++ code can do coordinate conversions, does not use an external library. − LSST C++ code does not need to do coordinate conversions and always uses TAI time and ICRS RA/Dec. − Use Astropy in Python code and very simple implementations in C++ – Integer nanoseconds for time. – Generic spherical coordinate.

8 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Tables − The LSST table library is optimized for efficient append operations using fixed table schemas. − The Astropy table class is optimized for column access and visualization. − Astropy have extended their constructor (in version 1.2) to accept objects that have a __astropy_table__ method providing a memory view compatible with the Astropy table data model. − LSST tables can now make use of more visualization tools and can be exported to pandas.

9 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Quantity − Very powerful Astropy facility to combine units with values. − Provide an additional check in an API, removing an entire class of bug. – Cannot combine quantities with incompatible units. − Down side is performance penalty. − LSST will adopt quantities on a case by case basis to gain experience in their use. − Have changed all unit strings in LSST codebase to be compatible with Astropy units (especially important for tables to allow columns to be interpreted as quantities). – Table units now validated.

10 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom C++ bindings − LSST use SWIG, Astropy prefer Cython. − C++ code balloons to 1.6 million lines when the 100k C++ interface is wrapped by SWIG. Compile times are dreadful. − Cython C++ support is lacking. − Pybind11 has much more potential (header only C++). − Actively experimenting switching SWIG to Pybind11 – Astropy project open to accepting bindings using this interface.

11 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Areas where Astropy and LSST can both benefit − Generalized World Coordinate Systems − Regions − Measurement Algorithms − N-Dimensional Data representations

12 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom World Coordinate Systems − FITS-WCS is not capable of representing all the distortions that will be present in LSST optics, camera hardware (tree rings, brighter-fatter…). − STScI have a similar problem with JWST instruments. − Need a scheme that allows mappings to be chained in parallel and series that can represent these different effects. − Starlink AST and STScI GWCS – Different strengths − Serialization format is key.

13 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Regions

14 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Regions: Footprints

15 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Measurement Algorithms − LSST will push the state of the art for algorithms dealing with source detection, source deblending and measurement. − Important that these algorithms are easily available to the community. − Currently depend on lots of LSST infrastructure classes and hard to disentangle. − Approach for now is to make them Astropy compatible but not necessarily standalone as Astropy Affiliated Packages.

16 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom N-Dimensional Data Representations − LSST uses an Exposure class to represent data, variance, quality masks, WCS, PSF, and exposure metadata.

17 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom

18 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom N-Dimensional Data Representations − LSST uses an Exposure class to represent data, variance, quality masks, WCS, PSF, and exposure metadata. − Astropy has an NDData that can do some of this but has evolved in a somewhat different direction. – A “pixel origin” needs to be added to NDData. − A duck-typed Exposure that can look like an NDData would be very useful, allowing LSST Exposures to be used by Astropy software packages.

19 SPIE Astronomical Telescopes + Instrumentation | 26 June - 1 July 2016 | Edinburgh, United Kingdom Summary − LSST are keen to be part of the Astropy community. − Have started to use Astropy classes in Python code. − Interoperability of LSST objects in Astropy is a priority. − Considering how best to contribute LSST code to the Astropy project (as Affiliated package and core contributions). − Lower the barrier to entry for people using the LSST software.