1. Overview and new developments in Herschel Data Processing
Stephan Ott Herschel Science Data Processing Development Manager Herschel Science Data Processing Coordinator Herschel Science Centre on behalf of all contributors of the Herschel mission

2. Overview Overview of Herschel Data Processing
Major improvements during the last year
What can HSC/NHSC and HIPE do for your Herschel Data?
Further plans and milestones
What can you do for HIPE?
Questions?
NGC 1999 observed by Herschel
ESA / PACS & SPIRE Consortium, Tom Megeath, NASA/JPL-Caltech/Univ. of Toledo

3. Overview of Herschel Data Processing
Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia

4. Overview of Herschel Data Processing
The system combines for the first time data retrieval, pipeline execution and scientific analysis in one single environment
All tools for data reduction and analysis, e.g. also the expert applications for e.g. instrument calibration are part of the Data Processing System. Therefore the community has access to the same system as the instrument experts
The Herschel Data Processing software is coded in Java/Jython to be license free and portable for different operating systems
Formal support is provided for Windows XP, Vista and Windows 7,Linux, Mac OS X 10.5 ("Leopard") and Mac OS X 10.6 ("Snow Leopard")
Herschel Science Centre (ESA), the Instrument Control Centres (HIFI, PACS and SPIRE) and NHSC jointly manage and contribute to the Herschel Data Processing System

5. Overview of Herschel Data Processing
Herschel Data Processing is a major project, with over 200 contributors and currently 60 full-time equivalents working on calibration, coding, documentation, quality control, testing and tutoring
Very distributed development – we work 24/5 (and often 24/7) on Herschel Data Processing System
300 data processing related software tickets are raised – and resolved – each month
A major HIPE version is released around each three months. This is currently HIPE 5.0
Minor HIPE versions (currently 5.2) are released around each three weeks. These serve also as the operational version to generate the standard products
All releases can be downloaded via
Latest HIPE developer releases are available via

6. Major Improvements during the last year

7. Improvements for user tools, HIPE and pipeline system
User friendliness and performance
Pre-installation steps for installer
Notification of updated user releases
HIPE perspectives have been simplified
Autocompletion in the console
Jobs can be shown and cancelled individually via status progress indicator
Option to gather user feedback
Restructuring of documentation
Support of plug-ins
Updates of documentation start-up page, links to data reduction guides from Help menu

8. Improvements for HIFI
Correct pointing information in datasets (aberration corrected) by use of ephemeris information
Source emission is now displayed as Antenna Temperature rather than flux
Noise reduction in Single Point Frequency Switch and Load/Chop modes
Purity of band 5b
Configurable HIFI pipeline
Spectrum Toolbox stabilisation (SpectrumExplorer, SpectrumFitter and all Spectral Tools integrated in HIPE)
Introduction of HIFI Cookbook and continued documentation updates

9. Improvements for SPIRE
Jump detection in signal timelines, allowing better temperature drift correction
New phase correction algorithm improves small-scale spectral shape
Improved spectral cube creation is now done with the photometer naive map-maker
Baseline correction/destriping algorithm. Work on this area is ongoing
Level 1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector
Level 2 spectral cubes are now created by averaging flux values within each pixel in the output map
Implementation of icons/browse products for the Herschel Science Archive

10. Improvements for SPIRE FTS products
HCSS 6.0, where better line detection is seen

11. What can HSC/NHSC and HIPE do for your Herschel Data?
Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia

12. Herschel Pipeline Processing and Data Quality Control
Pipelines are executed on the ESAC Grid to produce Herschel Products to different reduction levels
Level 0 raw data
Level 1 instrumental and satellite effects removed
Level 2 scientific analysis can be performed
The data are available in the Herschel Science Archive after the processing is finished (usually on the same day of reception of the data from the satellite)
The fast data quality control cycle by the Technical Assistants and Instrument Calibration Scientists takes a few days
data quality control is a combination of automatic screening and manual inspection
quality control reports are electronically distributed to experts

13. Bulk and on-demand reprocessing
Currently HSC performs a bulk reprocessing of all data taken up to now twice a year using an even HCSS version (HCSS 2, 4, 6, 8, …)
Bulk reprocessing with HCSS 4 is nearly complete
Significant improvements in GRID software and access to the Herschel Science Archive should lead to a much faster reprocessing with HCSS 6
Automatic quality control is performed using the latest quality control flags
Spot-checks are performed on a few observations
Users can request reprocessing of observations with the latest operational release using on-demand processing

14. Herschel Helpdesk
Contact the HSC helpdesk
or the NHSC helpdesk
for any questions, bug reports and suggestions you have regarding Herschel Data Processing (and beyond)

15. Further Plans and Milestones

16. Next Steps and Schedule
March 2011 Release of HIPE 6.0
March 2011 start of bulk reprocessing of all data taken up to now with HCSS 6
14 – 18 March 2011 Data Processing ESAC
Spring 2011 Release of eHSA 4.0
~ May 2011 Release of HIPE 7.0
~ August 2011 Release of HIPE 8.0
~ November 2011 Release of HIPE 9.0
~ March 2012 Release of HIPE 10.0

17. Foreseen improvements for user tools, HIPE and pipeline system
Reduce HIPE freezes and hangs
Improvement of calibration and data reduction pipelines
Improvement of user friendliness
Data manager for Herschel spectra to improve the robustness of spectrum explorer interface
Taskvariable association and useful tooltips
Simplified data input/output. Access to reprocessed data.
Addition of useful example scripts to the build
Harmonization of final spectra and VO. Workflow between Herschel products/HIPE and VO
Combination of data from the different Herschel instruments
Publication quality plots

18. Foreseen improvements for user tools, HIPE and pipeline system
Upgrade to Jython 2.5
Improved automatic testing to increase confidence in developer builds and speed-up delivery cycle for releases to the community
Improvement of code quality so the system will be maintainable with the reduced manpower we can expect during post-operations
Improvements to quality control framework, especially to support the bulk reprocessing exercises

19. Foreseen improvements for HIFI
Spectral maps (OTF and DBS Raster)
Inclusion of uplink information how the map has been carried out on the sky
Implementation of rotated maps (non-square RA / Dec grid)
Exporting the deconvolved HIFI spectrum to CLASS
Implementation of browse products and postcards for the Herschel Science Archive

20. Foreseen improvements for SPIRE
Baseline correction/destriping algorithm (photometer)
2nd level deglitching (photometer)
Correction of electrical crosstalk
zero point calibration of scan maps
Bright source mode pipeline (spectrometer)
Documentation how to perform end-to-end processing of calibrated full-range spectra with PACS and SPIRE

21. What can you do for HIPE?

22. Provide your feedback
… on documentation
… on HIPE
… on HIPE crashes

23. Reply to the DP Questionnaire
Participate in the Herschel Data Processing Interest Lists and User’s Group. Subscribe at
Participate in the HIPE user/developer workshops
Collaborate to HIPE development as contributor and tester
The worst feedback is the feedback not given

24. Your questions to me?