PPA Stack User Driven Image Stacking for ODI data via a Highly Customizable Web Interface Soichi Hayashi Indiana University - Pervasive Technology Institute.

Slides:



Advertisements
Similar presentations
MICHAEL MARINO CSC 101 Whats New in Office Office Live Workspace 3 new things about Office Live Workspace are: Anywhere Access Store Microsoft.
Advertisements

Chapter 12 User Interface Design
1 Software Engineering: A Practitioner’s Approach, 6/e Chapter 12b: User Interface Design Software Engineering: A Practitioner’s Approach, 6/e Chapter.
Edoclite and Managing Client Engagements What is Edoclite? How is it used at IU? Development Process?
Information and Business Work
DCS Architecture Bob Krzaczek. Key Design Requirement Distilled from the DCS Mission statement and the results of the Conceptual Design Review (June 1999):
15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.
These courseware materials are to be used in conjunction with Software Engineering: A Practitioner’s Approach, 6/e and are provided with permission by.
Panel Summary Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University XLDB 23-October-07.
Connecting Diverse Web Search Facilities Udi Manber, Peter Bigot Department of Computer Science University of Arizona Aida Gikouria - M471 University of.
© 2006 IBM Corporation SOA on your terms and our expertise Discovering the Value of SOA SOA In Action SOA & End-2-End Business Driven Development using.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
Building Data-intensive Pipelines Ravi K Madduri Argonne National Lab University of Chicago.
High-Speed, High Volume Document Storage, Retrieval, and Manipulation with Documentum and Snowbound March 8, 2007.
DYNAMICS CRM AS AN xRM DEVELOPMENT PLATFORM Jim Novak Solution Architect Celedon Partners, LLC
Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)
Architecture Of ASP.NET. What is ASP?  Server-side scripting technology.  Files containing HTML and scripting code.  Access via HTTP requests.  Scripting.
Crystal Hoyer Program Manager IIS Team Preview of features that will be announced at MIX09 Please do not blog, take pictures or video of session.
XCAT Science Portal Status & Future Work July 15, 2002 Shava Smallen Extreme! Computing Laboratory Indiana University.
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
Kuali Rice at Indiana University Rice Setup Options July 29-30, 2008 Eric Westfall.
GRACE Project IST EGAAP meeting – Den Haag, 25/11/2004 Giuseppe Sisto – Telecom Italia Lab.
Data Management Subsystem: Data Processing, Calibration and Archive Systems for JWST with implications for HST Gretchen Greene & Perry Greenfield.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI CloudBroker Platform integration into WS-PGRADE/gUSE Zoltán Farkas MTA.
SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline Language(s) We plan to develop CALCOS using Python and C Another programming language?
Magnetic Field Measurement System as Part of a Software Family Jerzy M. Nogiec Joe DiMarco Fermilab.
Customized cloud platform for computing on your terms ! Nirav Merchant
IPlant cyberifrastructure to support ecological modeling Presented at the Species Distribution Modeling Group at the American Museum of Natural History.
Portfolio v1.0 Products. Benefits Scalable Fast Full interface via web services Fully integrated with Microsoft SharePoint Easy navigation Competence.
Doug Tody E2E Perspective EVLA Advisory Committee Meeting December 14-15, 2004 EVLA Software E2E Perspective.
1 Sergio Maffioletti Grid Computing Competence Center GC3 University of Zurich Swiss Grid School 2012 Develop High Throughput.
1 Geospatial and Business Intelligence Jean-Sébastien Turcotte Executive VP San Francisco - April 2007 Streamlining web mapping applications.
DataNet – Flexible Metadata Overlay over File Resources Daniel Harężlak 1, Marek Kasztelnik 1, Maciej Pawlik 1, Bartosz Wilk 1, Marian Bubak 1,2 1 ACC.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
1 Computing Challenges for the Square Kilometre Array Mathai Joseph & Harrick Vin Tata Research Development & Design Centre Pune, India CHEP Mumbai 16.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Streamflow - Programming Model for Data Streaming in Scientific Workflows Chathura Herath.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
Prologue Major VO objective: “Data at your fingertips” (Lawrence) VO-user’s desire: “going backstage” –Data quality assessment by user (talk Gilmore)
1 COSC 4406 Software Engineering COSC 4406 Software Engineering Haibin Zhu, Ph.D. Dept. of Computer Science and mathematics, Nipissing University, 100.
What the Data Products Program Offers Users Todd Boroson Dick Shaw Presentation to NOAO Users Committee October 23, 2003.
A radiologist analyzes an X-ray image, and writes his observations on papers  Image Tagging improves the quality, consistency.  Usefulness of the data.
A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke.
6 February 2009 ©2009 Cesare Pautasso | 1 JOpera and XtremWeb-CH in the Virtual EZ-Grid Cesare Pautasso Faculty of Informatics University.
Lightweight construction of rich scientific applications Daniel Harężlak(1), Marek Kasztelnik(1), Maciej Pawlik(1), Bartosz Wilk(1) and Marian Bubak(1,
Ray Plante for the DES Collaboration BIRP Meeting August 12, 2004 Tucson Fermilab, U Illinois, U Chicago, LBNL, CTIO/NOAO DES Data Management Ray Plante.
Some comments on Portals and Grid Computing Environments PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce Computer Science, Informatics,
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
ESO Reflex A Graphical Workflow Engine for Data Reduction Richard Hook Euro VO Data Centres Alliance Theory & Grid Workshop, Garching, April 2008.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
Using the ARCS Grid and Compute Cloud Jim McGovern.
DSpace System Architecture 11 July 2002 DSpace System Architecture.
What is Firefly (1) A web UI framework for web applications
Ocean Observatories Initiative OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Scientific Workflows for OOI Ilkay Altintas Charles.
June 27-29, DC2 Software Workshop - 1 Tom Stephens GSSC Database Programmer GSSC Data Servers for DC2.
© Geodise Project, University of Southampton, Workflow Support for Advanced Grid-Enabled Computing Fenglian Xu *, M.
ICAT Status Alistair Mills Project Manager Scientific Computing Department.
Online Public Interface and Outreach ASTEP Website and Remote Science Collaboration Server 7/30/03.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI Accessing cloud resources through the WS-PGRADE/gUSE and CloudBroker integrated.
CyVerse Workshop Discovery Environment Overview. Welcome to the Discovery Environment A Simple Interface to Hundreds of Bioinformatics Apps, Powerful.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Convert generic gUSE Portal into a science gateway Akos Balasko.
EGI-InSPIRE RI EGI Compute and Data Services for Open Access in H2020 Tiziana Ferrari Technical Director, EGI.eu
Image Recognition Integration Server
Software Engineering: A Practitioner’s Approach, 6/e Chapter 12 User Interface Design copyright © 1996, 2001, 2005 R.S. Pressman & Associates, Inc.
Ramesh Baral Team: Marjani Peterson, Andre Guerrero
Module 01 ETICS Overview ETICS Online Tutorials
Gordon Erlebacher Florida State University
Presentation transcript:

PPA Stack User Driven Image Stacking for ODI data via a Highly Customizable Web Interface Soichi Hayashi Indiana University - Pervasive Technology Institute Co-Authors: Arvind Gopu, Michael Young (IU); Ralf Kotulla (University of Wisconsin, Milwaukee)

Soichi Hayashi Software Engineer at Indiana University / PTI - Research Technology / HTC

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven Download all images to laptop / local computing resources Run stacking application Flexible, but not scalable Increasingly difficult with data from ODI or LSST Low reproducibility Do-it-yourself

Raw Data Detrending (bias / dark / flat collections) Calibration (astrometric / photometric) Stacking Improve signal-to-noise ratio Reduce impact of stochastic image artifacts Science Data Observatory Driven No operator or user control, not scientifically useful Auto-generated?

Q. What can we do about stacking for ODI? Had to.. Simplify complex task of setting up the stacking workflow Fully customizable (flexible), but easy to use web UI Image processing done remotely on the cloud without downloading large amounts of data (at IU) Capable for performing complex stacking required

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive User searches for images to stack & creates a collection.

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Pre-sorts selected images into “likely” stack groups. Allows user to interact & customize stack groups / parameters Group images separated by <1deg RA/DEC (For each group) Create sub-groups based on FILTER type. Assign default parameters, *guess* best group name.

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Workflow executed on IU’s BigRed 2 podi_swarpstack Wrapper around SWarp application developed by Ralf Kotulla

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Ingest stacked images into ODI-PPA Archive

PPA Stack Search Collection Swarp UI BigRed 2 Stacked Image Data Archive Loop continues until user finishes analyzing / processing stacked images.

1.User search images to stack (m81)

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, etc.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, etc.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image. 5.You can review stacks parameters in a single page.

1.User search images to stack 2.Open SWarp UI using the search result. 3.Edit Stack options / (de)select images, for each stack. 4.Or.. (de)select stacks to be part of for each image. 5.You can review stacks parameters in a single page. 6.Submit job, sit back and relax!

M1 H_alpha Dither Sequence

M1 H_alpha Stack Median combine, background subtract, preserve extended objects with size < 3’ Processing time: ~70s

3-color image O III H alpha SDSS u

PPA Stack – Behind the Scene… Implemented as a Single-Page-Application (SPA) Fluid / Interactive UX Reduces the amount of page navigations and form submissions

PPA Stack – Behind the Scene… Implemented as a Single-Page-Application (SPA) Fluid / Interactive UX Reduces the amount of page navigations and form submissions AngularJS simplifies / speed up implementation.

PPA Stack – Behind the Scene… Microservice Architecture Independently deployable loosely-coupled components Component written in Java / Python / PHP / NodeJS Joined together by REST API & AMQP Messaging Bus

PPA Stack – Behind the Scene… Microservice Architecture Independently deployable loosely-coupled components Component written in Java / Python / PHP / NodeJS Joined together by REST API & AMQP Messaging Bus Minimize impact of adding new functionalities Easier to handle HA / scalability

Future Work Improve alignment between different stacks Multi-band photometry Continuum-subtract Narrow-band images (H-alpha, [O III], etc.) Allow user to convert set of raw images directly into stacked images Raw Images (Dither Pattern) -> Quick Reduce -> PPA Stack -> Archive Integrate other stacking applications (e.g. Montage) Many Others…

Contact ODI-PPA Project Manager Arvind Gopu Presenter / Software Engineer Soichi Hayashi Check out related posters! PPA Chart poster by Mike Young, et al. Pipeline Operator Interface poster by Wilson Liu, et al.

ODI-PPA – Partner Organizations ODI-PPA is a collaboration of the following organizations Pervasive Technology Institute (PTI) and UITS Research Technologies (RT) + IU Astronomy Capitalize on the expertise of PTI members who have led the effort to provide scientists in many different fields with user-friendly access to super-computing facilities IU Astronomy Expertise, Feedback University of Wisconsin (Astronomy) Build on the experience of Python based pipeline development experience. NOAO Science Data Management group Build on the experience of SDM and the legacy of IRAF and NHPPS Pipeline system WIYN Experience running telescopes, and supporting Astronomy scientific community