1. HST and JWST Pipelines and Reference Files
Rosa I Diaz
Sept 29, 2016

2. Strategic Plan Missions Innovation Community
Maintain HST (JWST) as the world’s leading observatory as long as its science return is competitive with other facilities.
Develop and support the science program of the James Webb Space Telescope on behalf of the community, with a focus of maximizing this observatory’s science productivity in its first 5-years
Operate a world-class archive and in partnership with our Universities and Centers, maximize the science return of community-based astronomy projects
Innovation
Pursue creative and innovative solutions to science operations challenges
Use new technology to increase access to astronomy data and information
Community
Serve as a trusted advocate of the astronomy community to NASA

3. Pipelines Two type of pipelines
Pipeline or Data Management System (DMS) Pipeline
Calibration Pipeline

4. DMS pipeline Data that comes from the spacecraft
Ingested by the DMS pipeline (telemetry conversion)
Using Engineering data
Using Planning Data
Using flight Observation logs
Needs to know how the data is taken and stored in the spacecraft.
Each type of data needs to be transformed into different types of FITS files

5. DMS pipeline (cont.) Processed by the Calibration Pipeline
Products delivered to the archive
Products accessible to General Observers (primary investigator group immediately) and public or other scientist after proprietary period

6. Calibration Pipeline
Pipeline that Calibrates the science data according to the algorithms or methods provided by the instrument teams (e.g. dark and bias subtraction, linearity correction etc.)
Produces data that is science ready
Instrument Teams provide with Algorithms
Science Support Branch (SSB) develops the pipeline

7. Calibration Pipelines
HST
Each instrument has its own calibration pipeline
Evolved separately
DMS/SSB support is independent
Managing updates is independent for each team
Active instruments: ACS, COS, WFC3, STIS
Heritage Instruments: NICMOS, FOS, FOC, HRS, HSP, WFPC, WFPC2, NICMOS

8. HST Calibration Pipelines
On-the-fly reprocessing is now being replaced by static archive
On-the-fly: Every time a dataset was requested it was re-processed
Static archive implies less processing of data:
When new calibration software is delivered
When new reference files affecting the observing mode, are delivered.
e.g. STIS, MAMA, E140H mode
Allows for faster delivery of archival data

9. JWST Pipeline
All the instruments have a common core pipeline (e.g. CALDETECTOR1).
Algorithms the same for most instruments and modes
Some specific steps for MIRI
Order might be different
Stage 1
CALDETECTOR1
Stage 2
CALIMAGE2
CALSPEC2
Stage 3
CALIMAGE3
CALCORON3
CALAMI3
CALTSO3
CALSPEC3

10. JWST Calibration Pipeline

11. Benefits of common code
Knowledge transferred from team to team
Different approaches considered
Software developed by a single group
More people available to update/develop software
Broader system design
Ability to perform team reviews for code development
Cross training

12. Calibration software & Reference Files
Calibration Pipelines use reference files for some of the steps
Reference files have to work with software
Reference files have to be
Instrument Teams (using calibration specific obsevations) Create Reference Files
Validates
Test
Delivers
ReDCaT Management Team
(Reference Data for Calibrations and Tools)
Review Validation
CRDS
(Calibration Reference Data System)
Certify
DMS
Pipeline
Process Data
New
Old affected
Archive
Users retrieval
Remote sites
Local
Use internally
For HST the ReDCaT Team

13. CRDS For HST: https://hst-crds.stsci.edu/
For JWST:
Files can be retrieved via the archive
Files can be retrieved via CRDS command line

17. Pipeline Changes and Updates
Changes to the Calibration Pipeline and sometimes to reference files are not a simple process
These changes might affect more than one instrument (JWST) or observing mode and have to be tested before delivered
DMS Pipeline testing includes:
Installation of new software
Test readiness review
Unit testing
Combine software and test as a group (integration testing)
Validation testing
Regression testing by INS
Performance testing

18. Process to update Cal Pipeline
Inst Team request change
Build
YES
Yes
SSB codes Change
Yes
DMS signoff No
INS Team Test Passes
SSB Test Passes No
Yes
Regression Test No

19. INS Request a new observing mode
Adding a new mode
Flight Software
APT
INS Request a new observing mode
CRDS
Yes
ETC
Archive
DMS Pipeline

20. Adding a new Reference File
SSB
Pysynphot/ETC
CRDS
INS defines a new reference file
DMS Pipeline
Archive