Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #1 PACS Overview and the latest developments in Herschel Data Processing Stephan Ott Herschel Science Data Processing Development Manager Herschel Science Data Processing Coordinator Herschel Science Centre on behalf of all contributors of the Herschel mission

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #2 PACS Overview Overview of Herschel Data Processing Major improvements during the last year What we and HIPE can do for the Herschel community Planned improvements and milestones What you can do for the Herschel community Questions? Aquila / W40 observed by Herschel ESA and the SPIRE & PACS consortia, Ph. André (CEA Saclay) for the Gould’s Belt Key Programme Consortia

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #3 PACS Eagle Nebula, credits ESA/PACS& SPIRE Consortium, Tracey Hill, Frédérique Motte, Laboratoire AIM Paris - Saclay, CEA/IRFU - CNRS/INSU - Uni. Paris Diderot, HOBYS Key Programme Consortia Overview of Herschel Data Processing

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #4 PACS Overview of Herschel Data Processing The system combines for the first time data retrieval, pipeline execution and scientific analysis in one single environment All tools for data reduction and analysis, e.g. also the expert applications for e.g. instrument calibration are part of the Data Processing System. Therefore the community has access to the same system as the instrument experts The Herschel Data Processing software is coded in Java/Jython to be license free and portable for different operating systems Formal support is provided for Windows XP, Vista and Windows 7,Linux, Mac OS X 10.5 ("Leopard") and Mac OS X 10.6 ("Snow Leopard") Herschel Science Centre (ESA), the Instrument Control Centres (HIFI, PACS and SPIRE) and NHSC jointly manage and contribute to the Herschel Data Processing System

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #5 PACS Overview of Herschel Data Processing Herschel Data Processing is a major project, with over 200 contributors and currently 60 full-time equivalents working on calibration, coding, documentation, pipeline operations, quality control, testing and tutoring Very distributed development – we work 24/5 (and often 24/7) on Herschel Data Processing System 300 data processing related software tickets are raised – and resolved – each month A major HIPE version is released around each six months. This is currently HIPE 8.0 Minor HIPE versions are released each few weeks. HIPE 8.1 was just released. These serve also as the operational version to generate the standard products All HIPE user releases can be downloaded via Latest HIPE developer releases are available via You are about to enter the Continuous Integration Build (CIB) system pages. These pages contain instances of HIPE in different stages of development. They have not been fully tested and may be unstable and/or unsuitable for your purposes. Some important functionality may be missing from these builds.

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #6 PACS Major improvements during the last year

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #7 PACS Highlights of user tools, HIPE and pipeline system improvements Reduction of crashes and freezes User friendliness (task-variable association and tooltips) Upgrade of Jython (HIPE’s interpreter) from 2.1 to Shielding installer against corruption due to network problems Support for DS9 to display and process Herschel images More protection against problematic plug-ins. You can now disable all plug-ins without having to start HIPE All SED models for asteroid observations have been added to the ESAC database Revamp of organisation of HIPE Owner’s Guide Creation of a YouTube channel of videos tutorials and a Twitter channel of daily tips:

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #8 PACS First connect to the VO in DS9 You can now see DS9 appear in the Send To menu in HIPE The resulting image after sending it to HIPE by clicking "SAOImage DS9" in the Send To menu DS 9 support

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #9 PACS App to access Herschel Quick Look Products for Android and iPhone

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #10 PACS Highlights of HIFI improvements  40% noise reduction and improved baseline quality for level 1 products of all observations performed in bands 6 and 7  Browse products for simple spectra  HIFI pipeline task GUI  For strong continuum sources an optional pipeline step can reduce or eliminate standing waves  Updated browse images for point mode observations  The HIFI User's Manual changed name to HIFI Data Reduction Guide, chapters were rearranged into more logical order  Addition of several chapters (introduction, chapter, quick start guide, DBS Point Mode cookbook, “How to flag and remove flags from your data” and Unit Conversion)

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #11 PACS HCSS 7 HCSS 8 Highlights of HIFI improvements 40% noise reduction in HIFI pipeline products

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #12 PACS Highlights of PACS improvements General –Improved viewers for inspecting / masking datapoints –Improved saving intermediate results on standard desktop machines with little memory Spectrometer –Interactive reduction scripts for all observing modes –Flatfielding for spectrometer interactive reduction scripts –Background normalisation method for chop/nod observations leads to improved flux calibration –Error propagation for Level 2 PACS rebinned cubes product & error bars for significance tests of line detections –Improvements in unchopped mode pipeline Photometer –Transient correction for miniscan map –Cross-talk correction for red channel Photometer and spectrometer browse products

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #13 PACS Transient correction for miniscan map Highlights of PACS improvements Photometer pipeline Cross-talk correction for red channel

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #14 PACS Spurious structures removed using the upgraded unchopped mode spectrometer pipeline Spurious structure Highlights of PACS improvements

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #15 PACS Highlights of SPIRE improvements General –Signal jump detector now handles "cooler burps" –Masking of data at non-nominal s/c velocities Photometer: –Creation of Level 2.5 products, combining several observations into one mosaic –Baseline removal and destriper tasks now fully available –SCalPhotChanRelGain calibration product to improve extended emission calibration Spectrometer –Pipeline processing of bright source mode observations with the SPIRE Spectrometer –Improvement of SCalSpecTeleRsrf to reduce the noise level in the RSRF Provision of Photometer (e.g. destriper, moving object correction) and Spectrometer scripts (e.g. background subtraction, line fitting) Provision of quick start guides; revision of the data reduction guide

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #16 PACS Improvements for SPIRE photometric products Improved handling of “cooler burps” in pipeline HCSS 7.0 HCSS 8.0

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #17 PACS Improvements for SPIRE FTS products Spectrometer pipeline product of bright mode observation

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #18 PACS What we and HIPE can do for the Herschel community

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #19 PACS Herschel Pipeline Processing and Data Quality Control Pipelines are executed on the ESAC Grid to produce Herschel Products to different reduction levels –Level 0raw data –Level 1instrumental and satellite effects removed –Level 2scientific analysis can be performed. For many instrument modes we achieved a very high pipeline product quality Products are available in the Herschel Science Archive after the processing is finished (usually on the same day of reception of the data from the satellite) Fast data quality control cycle by the Technical Assistants and Instrument Calibration Scientists takes a few days –data quality control is a combination of automatic screening and manual inspection –quality control reports are electronically distributed to experts

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #20 PACS Example of the quality of pipeline products: SPIRE photometry Multiple observations are combined into a level 2.5 product, eliminating artefacts …

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #21 PACS Combined observation (detail) Single observation (detail) Example of the quality of pipeline products: SPIRE photometry

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #22 PACS … getting a bigger field Example of the quality of pipeline products: SPIRE photometry Galactic Centre

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #23 PACS Example of the quality of pipeline products: SPIRE photometry … getting a better signal to noise ratio Credits: HerMES Oliver et al in prep. & SPIRE consortium

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #24 PACS You can access the Herschel Science Archive via It contains currently around publicly accessible observations HSC performs a bulk reprocessing of all data taken up to now twice a year Bulk reprocessing with HCSS 6.1 was completed five months ago Bulk reprocessing with HCSS 8.2 will start in a few weeks Automatic quality control is performed using the latest quality control flags Spot-checks are performed on a few observations Users can request reprocessing of observations with the latest operational release using on-demand processing Herschel papers using archival data were published already! These are great news – please continue to exploit the Herschel data to their fullest extent! Herschel Science Archive, bulk and on- demand reprocessing

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #25 PACS Planned improvements and milestones

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #26 PACS Next Steps and Schedule March 2012Start of bulk reprocessing of all data taken up to now with HCSS 8.2 Spring 2012Release of eHSA 4.0 June 2012Completion of bulk reprocessing June 2012Release of HIPE 9.0 Summer 2012HIPE Forum 2012 – Interaction meeting for Calibration Scientists, Developers and Expert users December 2012Release of HIPE 10.0

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #27 PACS Foreseen improvements for user tools, HIPE and pipeline system Common look and feel for the spectrum explorer and cube analysis toolbox Image analysis perspective Easing the data access via MyHSA Improvement of handling of ASCII tables Mechanism to notify users on removed names (tasks, functions...), why they were removed, and their replacement Smaller memory footprint for HIPE editor Speed-up of HIPE start-up Progress bar informs on progress of for-loops

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #28 PACS Foreseen improvements for user tools, HIPE and pipeline system Human-made indexes added to the core manual (HIPE Owner's Guide, Scripting Guide, Data Analysis Guide) Possibility to leave comments at the bottom of any page in the HIPE Help System Support to OSX Lion TBD: Easy access to calibration data, including version and updates TBD: Improvements of search capability of documentation system TBD: Publication ready plots

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #29 PACS Foreseen improvements for HIFI Scripts to permit fringe removal / current matching HIFI level 2.5 product to hold data of Spectral Surveys TBD: Browse products for spectral surveys in maps

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #30 PACS Foreseen improvements for PACS Photometer –Addressing further specific cases in IPipe scripts Spectrometer –Broadband reliability –Improved spectral pixel flatfield broad ranges (SED) –Correction Flux loss due to pointing Convolution kernels for photometer and spectrometer maps aiding the interpretation of extended and point sources TBD: Reduction in memory consumption

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #31 PACS Foreseen improvements for SPIRE Common (both Photometer and Spectrometer) –Improved processing speed using multi-threading –improved task for correction of "cooler burps" Photometer –Pipeline Processing kernel-smoothed maps (aka "drizzling") improved error maps (possibly as an option) improved baseline correction using "destriper“ –Interactive processing source timeline fitter task improvements to bolometer finder tool selection of turnaround data in map creation

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #32 PACS Foreseen improvements for SPIRE Spectrometer –Pipeline processing smaller size for level-1 products –Improved calibration and data processing updated point source calibration (improved Uranus model) updated bright mode calibration (better PCAL module) improved calibration for Low Resolution observations (hopefully improved calibration, but at minimum, updated advice) more straightforward calling sequence for phase-correction task corrected spectral averaging - more flexible usage –Interactive processing drag and drop from SDS Explorer detector layout to variables tab new and updated useful scripts for background subtraction and spectral noise

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #33 PACS What you can do for the Herschel community

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #34 PACS Provide your feedback … on documentation … on HIPE … on HIPE hangs and crashes

Join the DP Interest Lists via where all DP-related events are announced Join the DP telecons to discuss special topics of data reduction for the instruments Join the HIPE Community users’ portal and contribute with questions and answers Join the HIPE development as contributor Join the NHSC DP webinars organised this spring Join the testing campaigns of new versions

Data Processing Workshop ESAC, Madrid, Spain 20 – 24 February 2012 Stephan Ott VG #36 PACS Your questions for me?