PPA Stack User Driven Image Stacking for ODI data via a Highly Customizable Web Interface Soichi Hayashi Indiana University - Pervasive Technology Institute Co-Authors: Arvind Gopu, Michael Young (IU); Ralf Kotulla (University of Wisconsin, Milwaukee)

Soichi Hayashi Software Engineer at Indiana University / PTI - Research Technology / HTC

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven Download all images to laptop / local computing resources Run stacking application Flexible, but not scalable Increasingly difficult with data from ODI or LSST Low reproducibility Do-it-yourself

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven No operator or user control, not scientifically useful Auto-generated?

Q. What can we do about stacking for ODI? Had to.. Simplify complex task of setting up the stacking workflow Fully customizable (flexible), but easy to use web UI Image processing done remotely on the cloud without downloading large amounts of data (at IU) Capable for performing complex stacking required

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive User searches for images to stack & creates a collection.

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Pre-sorts selected images into “likely” stack groups. Allows user to interact & customize stack groups / parameters Group images separated by <1deg RA/DEC (For each group) Create sub-groups based on FILTER type. Assign default parameters, *guess* best group name.

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Workflow executed on IU’s BigRed 2 podi_swarpstack Wrapper around SWarp application developed by Ralf Kotulla

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Ingest stacked images into ODI-PPA Archive

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Loop continues until user finishes analyzing / processing stacked images.

1.User search images to stack (m81)

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, etc.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, etc.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image. 5.You can review stacks parameters in a single page.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image. 5.You can review stacks parameters in a single page. 6.Submit job, sit back and relax!

M1 H_alpha Dither Sequence

M1 H_alpha Stack Median combine, background subtract, preserve extended objects with size < 3’ Processing time: ~70s

3-color image O III H alpha SDSS u

PPA Stack – Behind the Scene… Implemented as a Single-Page-Application (SPA) Fluid / Interactive UX Reduces the amount of page navigations and form submissions

PPA Stack – Behind the Scene… Implemented as a Single-Page-Application (SPA) Fluid / Interactive UX Reduces the amount of page navigations and form submissions AngularJS simplifies / speed up implementation.

PPA Stack – Behind the Scene… Microservice Architecture Independently deployable loosely-coupled components Component written in Java / Python / PHP / NodeJS Joined together by REST API & AMQP Messaging Bus

PPA Stack – Behind the Scene… Microservice Architecture Independently deployable loosely-coupled components Component written in Java / Python / PHP / NodeJS Joined together by REST API & AMQP Messaging Bus Minimize impact of adding new functionalities Easier to handle HA / scalability

Future Work Improve alignment between different stacks Multi-band photometry Continuum-subtract Narrow-band images (H-alpha, [O III], etc.) Allow user to convert set of raw images directly into stacked images Raw Images (Dither Pattern) -> Quick Reduce -> PPA Stack -> Archive Integrate other stacking applications (e.g. Montage) Many Others…

Contact ODI-PPA Project Manager Arvind Gopu Presenter / Software Engineer Soichi Hayashi Check out related posters! PPA Chart poster by Mike Young, et al. Pipeline Operator Interface poster by Wilson Liu, et al.

ODI-PPA – Partner Organizations ODI-PPA is a collaboration of the following organizations Pervasive Technology Institute (PTI) and UITS Research Technologies (RT) + IU Astronomy Capitalize on the expertise of PTI members who have led the effort to provide scientists in many different fields with user-friendly access to super-computing facilities IU Astronomy Expertise, Feedback University of Wisconsin (Astronomy) Build on the experience of Python based pipeline development experience. NOAO Science Data Management group Build on the experience of SDM and the legacy of IRAF and NHPPS Pipeline system WIYN Experience running telescopes, and supporting Astronomy scientific community