COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 12 Science Data Processing

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 2 of 12 OPUS Science Data Processing

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 3 of 12 OPUS Processes  Data Partitioning – segments the telemetry stream into standard EDT dataset – appends multiple dumps into a single exposure file > uses an index file to avoid sorting packets > flags packet errors caught by PACOR  Support Schedule – gathers proposal information from PMDB – test proposals required for development > test version of PMDB must be populated by TRANS > Thermal Vac support schedule required for full processing

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 4 of 12 OPUS Processes (cont.)  Data Validation – decodes the exposure and engineering parameters in the telemetry and compares them to the planned values – internal header specification (from Ball) > PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 [DONE] – Planned value comparisons (“flags and indicators”) need to be determined by the Instrument Scientists  World Coordinate System – implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing – aperture positions must be defined

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 5 of 12 OPUS Processes (cont.)  Generic Conversion – Generic Conversion outputs uncalibrated data > rawtag (Time-Tag), rawimage, pha, asn, spt, trl – data will be output in standard FITS format with image or table extensions – primary header will contain keywords inherited by all extensions and a null data array – Extensions > Time-Tag mode will be FITS binary tables (events + good times intervals) > ACCUM mode will be images grouped by imsets consisting of science array, data quality array, and null error array. > FUV data will contain separate files for each detector (a and b) > data quality array will be null if no telemetry dropouts

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 6 of 12 OPUS Processes (cont.)  Generic Conversion (cont.) – Required for development > DM-06 to develop algorithms for data formatting [DONE] > keyword definitions (ICD-19) are currently being worked by the Instrument Scientists world coordinate definitionsworld coordinate definitions exposure time calculationsexposure time calculations calibration switches and selection criteriacalibration switches and selection criteria calibration file name keywordscalibration file name keywords

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 7 of 12 OPUS Processes (cont.)  Data Collector – OPUS will ensure all component exposures are present before processing further > rules for processing incomplete associations have been defined – association table contains information about the product dataset > dataset self-documenting – only associations required for data processing will be constructed in the OPUS pipeline > FP splits > Repeat obs

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 8 of 12 Calibration  OPUS will use STSDAS calibration software  run on ALPHA / Tru64 UNIX platform in operations  expands size of dataset – converts integer raw data to real – possible expansion of data quality array for ACCUM mode – expansion of the error array.  Need calibration reference files for testing (at least dummies)

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 9 of 12 Other Science Data Modes  requirements for data content of each of these other science data modes must be defined by Instrument Scientists – target acquisitions – microprocessor memory dump – engineering diagnostic data – cumulative detector image > MAMA cumulative image like STIS for NUV > cumulative image for FUV detector

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 10 of 12 CODE REUSE  Core OPUS system (OPUS 12.1) – ~236,000 lines of code – 100% reuse  COS specific processes – Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Pipeline”, > 5076 lines of code > 71% reuse of existing OPUS modules  Expect > 99% reuse of existing data processing software for COS, based on lines of code. – All SI complexity contained in relatively few lines of code.  Efficient use of existing system!

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 11 of 12 Major Science Data Processing Requirements Summary  Internal header specification – DM-06 to document content and format of science internal header – PDB (EUDL.DAT, TDFD.DAT) defined and fully populated  Keyword definitions  Planned values “flags and indicators” for Data Validation  Aperture definitions

COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 12 of 12 Test Data Requirements  Test data from detectors on ops bench expected in November 2001 and from integrated instrument in March 2002  Test data to be provided by IDT/Instrument Scientists and Engineers should include all science and engineering modes  Test data must include – PMDB population and PDB definition – spectra that enable OPUS to determine proper orientation – list of possible error conditions to simulate – data that simulate error conditions – enough data for throughput test