HST and JWST Pipelines and Reference Files Rosa I Diaz Sept 29, 2016

Strategic Plan Missions Innovation Community Maintain HST (JWST) as the world’s leading observatory as long as its science return is competitive with other facilities. Develop and support the science program of the James Webb Space Telescope on behalf of the community, with a focus of maximizing this observatory’s science productivity in its first 5-years Operate a world-class archive and in partnership with our Universities and Centers, maximize the science return of community-based astronomy projects Innovation Pursue creative and innovative solutions to science operations challenges Use new technology to increase access to astronomy data and information Community Serve as a trusted advocate of the astronomy community to NASA

Pipelines Two type of pipelines Pipeline or Data Management System (DMS) Pipeline Calibration Pipeline

DMS pipeline Data that comes from the spacecraft Ingested by the DMS pipeline (telemetry conversion) Using Engineering data Using Planning Data Using flight Observation logs Needs to know how the data is taken and stored in the spacecraft. Each type of data needs to be transformed into different types of FITS files

DMS pipeline (cont.) Processed by the Calibration Pipeline Products delivered to the archive Products accessible to General Observers (primary investigator group immediately) and public or other scientist after proprietary period

Calibration Pipeline Pipeline that Calibrates the science data according to the algorithms or methods provided by the instrument teams (e.g. dark and bias subtraction, linearity correction etc.) Produces data that is science ready Instrument Teams provide with Algorithms Science Support Branch (SSB) develops the pipeline

Calibration Pipelines HST Each instrument has its own calibration pipeline Evolved separately DMS/SSB support is independent Managing updates is independent for each team Active instruments: ACS, COS, WFC3, STIS Heritage Instruments: NICMOS, FOS, FOC, HRS, HSP, WFPC, WFPC2, NICMOS

HST Calibration Pipelines On-the-fly reprocessing is now being replaced by static archive On-the-fly: Every time a dataset was requested it was re-processed Static archive implies less processing of data: When new calibration software is delivered When new reference files affecting the observing mode, are delivered. e.g. STIS, MAMA, E140H mode Allows for faster delivery of archival data

JWST Pipeline All the instruments have a common core pipeline (e.g. CALDETECTOR1). Algorithms the same for most instruments and modes Some specific steps for MIRI Order might be different Stage 1 CALDETECTOR1 Stage 2 CALIMAGE2 CALSPEC2 Stage 3 CALIMAGE3 CALCORON3 CALAMI3 CALTSO3 CALSPEC3

JWST Calibration Pipeline

Benefits of common code Knowledge transferred from team to team Different approaches considered Software developed by a single group More people available to update/develop software Broader system design Ability to perform team reviews for code development Cross training

Calibration software & Reference Files Calibration Pipelines use reference files for some of the steps Reference files have to work with software Reference files have to be Instrument Teams (using calibration specific obsevations) Create Reference Files Validates Test Delivers ReDCaT Management Team (Reference Data for Calibrations and Tools) Review Validation CRDS (Calibration Reference Data System) Certify DMS Pipeline Process Data New Old affected Archive Users retrieval Remote sites Local Use internally For HST the ReDCaT Team

CRDS For HST: https://hst-crds.stsci.edu/ For JWST: https://jwst-crds.stsci.edu/ Files can be retrieved via the archive Files can be retrieved via CRDS command line

Pipeline Changes and Updates Changes to the Calibration Pipeline and sometimes to reference files are not a simple process These changes might affect more than one instrument (JWST) or observing mode and have to be tested before delivered DMS Pipeline testing includes: Installation of new software Test readiness review Unit testing Combine software and test as a group (integration testing) Validation testing Regression testing by INS Performance testing

Process to update Cal Pipeline Inst Team request change Build YES Yes SSB codes Change Yes DMS signoff No INS Team Test Passes SSB Test Passes No Yes Regression Test No

INS Request a new observing mode Adding a new mode Flight Software APT INS Request a new observing mode CRDS Yes ETC Archive DMS Pipeline

Adding a new Reference File SSB Pysynphot/ETC CRDS INS defines a new reference file DMS Pipeline Archive