1. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline and Archives Preliminary Design Review 7 December 2000

2. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline and Archives Review Panel Gerard Kriss – Chair Ken Brownsberger Ray Kutina Richard Shaw cos_review_panel@stsci.edu

3. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA Agenda Welcome and IntroductionW. King 9:00 COS OverviewT. Keyes 9:05 COS Dataset FormatT Keyes 9:20 Post-SM4 Science Data VolumeT. Keyes10:00 COS Association FormatE. Hopkins10:20 BREAK10:45 CALCOS P. Hodge11:00 OPUS/OTFRD. Swade11:20 Hubble Data ArchiveW. King11:50 Schedule and ResourcesW. King12:05 ClosingW. King12:15

4. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline PDR Instrument Overview Charles D. (Tony) Keyes 7 December 2000

5. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 5HDA COS Design   Moderate Resolution (R~20,000) point-source spectroscopy; 0.1 arcsec pointing; 15 km/sec absolute (5 km/sec relative) radial velocity accuracy   Highest Possible Throughput   Maximize wavelength coverage per exposure   FUV: – – single reflection system – – large format solar-blind detector   NUV: – – scanning system, enhance wavelength coverage through use of three camera optics

6. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 6HDA COS Detectors   FUV – – Cross-Delay Line (XDL) detector > > two 85 mm (dispersion-direction) x 10 mm segments (~10 mm gap between segments) > > 2 x 16,384 x 1024 pixels; ~7 pixels per resolution element along dispersion > > “stim pulses” to characterize stretching and shifting in both coordinates   NUV – – MAMA (STIS flight spare) > > 25 mm x 25 mm detector format > > 1024 x 1024 pixels; no subarrays; 2 pixels per resolution element

7. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 7HDA COS Operating Characteristics   Both channels can not be operated simultaneously   Either one or both FUV segments can be used   TIME-TAG will be prime operating mode – – Full-frame readout - both FUV and NUV – – reduce data volume   ACCUM mode – – NUV: full-frame readout (1 M pixels) – – FUV: partial-frame readout [full dispersion-direction dimension by 256 pixels readout cross-dispersion] (4 M pixels per segment)

8. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 8HDA COS Optics   Apertures – – Primary Science Aperture (PSA): 2.5˝ circular field stop – – Bright Object Aperture (BOA): 2.5˝ + ND2 circular field stop – – Wavelength Calibration Aperture (WCA) – – Flat Field Aperture (FFA)   Optical Elements (Gratings, Mirrors) – – FUV: G130M, G160M, G140L > > ~8 pre-defined settings (2 M or 1L to secure full spectral coverage) – – NUV: G185M, G225M, G265M, G230L, TA1 mirror > > ~30 pre-defined settings > > ~3˝ field may be imaged with mirror

9. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 9HDA COS Modes 1650-3200

10. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 10HDA COS Association Overview   Each spectroscopic science exposure will have at least 1 wavecal associated with it, as for STIS – – wavecals may be shared by different science exposures   Only science/wavecal groupings, FP-SPLITs, and REPEATOBS will be formally associated in pipeline processing and in HDA   COS associations will generally follow NICMOS design in having all associated observations in separate datasets which are linked via an association table

11. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 11HDA COS Exposure Type Summary

12. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 12HDA COS Pipeline Special Goal   Be capable of processing uncalibrated observations, cataloging, and archiving data by date of COS Thermal Vac – – no associations in Thermal Vac capability – – no calcos in Thermal Vac capability

13. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline PDR File Structure Overview Charles D. (Tony) Keyes 7 December 2000

14. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 14HDA COS Data Format  2-d data: FITS files containing imsets consisting of science and data quality images (unlike STIS, no error array)  1-d data and TIME-TAG lists: FITS tables containing fluxes, wavelengths, errors, quality, positions, and other related quantities  May explore compression for reduced dataset volume  Data from each FUV detector segment stored as separate imset series within a single file – All associated data (e.g., wavecals, FP-SPLIT components, REPEATOBS components, STIMs) use separate files (unlike STIS)

15. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 15HDA Data File Naming Conventions   Product Suffix Type Contents   ---------------------------------------------------------------   Uncalibrated _raw image Raw counts image   _rtg table Raw TIME-TAG event list   _asn table Association file   _trl table Trailer file (input)   _lrc image Local rate check image [TBD]   _lsp text LRC support file (header) [TBD]   _jit table   _jif image   _pha table Pulse-height (FUV ACCUM only)   _rs1 image Stim-pulse 1 (FUV ACCUM only)   _rs2 image Stim-pulse 2 (FUV ACCUM only)  

16. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 16HDA Data File Naming Conventions (continued)   Product Suffix Type Contents  ---------------------------------------------------------------   Calibrated _ctg table Calibrated TIME-TAG event list   _flt image Flat-fielded, corrected counts   _fls image Summed flat-fielded corrected-counts   _x1d table 1-D extracted spectra   _x1s table Summed (combined) 1-D extracted spectra)   _cs1 image Corrected stim-pulse 1 (FUV ACCUM only)   _cs2 image Corrected stim-pulse 2 (FUV ACCUM only)   _trl table Trailer file (output)   _spt image Support file (header)  ---------------------------------------------------------------  Wavecal exposures produce same calibrated files as science, except there may be no flux calibration applied to the _x1* files (TBD). Dark and flat exposures produce no _x1* files. Acquisition exposures produce no calibrated products.

17. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 17HDA COS Observational Quantity Definitions   x = raw detector x-position (I*2)   y = raw detector y-position (I*2)   t = time of photon detection (R*4)   ph = pulse-height (I*1)   xcorr = corrected detector x-position (R*4)   ycorr = corrected detector y-position (R*4)   EPS = photon list sensitivity-parameter (R*4)   obsC = observed counts (R*4)   C = corrected counts (R*4)   q = data quality (I*1)   lambda = wavelength (R*4)   F = calibrated flux (F-lambda) (R*4)   G = gross extracted counts (R*4)   N = net extracted counts (R*4)   B = background extracted counts (R*4)   S = sensitivity function (R*4)

18. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 18HDA File Contents: Observational Quantities   _rtg: contains x,y,t,ph (no ph for NUV) – – FUV: 9 bytes/event; NUV: 8 bytes/event   _ctg: contains xcorr,ycorr,t,ph,EPS,q,lambda (no ph for NUV) – – FUV: 22 bytes/event; NUV: 21 bytes/event   _raw: contains obsC – – FUV: 32 Mbyte/file; NUV: 4 Mbyte/file   _flt: contains C,q (in each imset) – – TIME-TAG: FUV: 160 Mbyte/file; NUV: 5 Mbyte/file – – ACCUM: FUV: 40 Mbyte/file; NUV: 5 Mbyte/file   _fls: same as _flt   “_obs”: contains obsC (in each imset) [size same as _raw]   _x1d: contains F,lambda,G,N,B,S,q (in each imset) – – FUV: ~1 Mbyte/file; NUV: ~0.08 Mbyte/file   _x1s: same as _x1d   _rs1: contains obsC (same for _rs2)   _cs1: contains C (same for _cs2)

19. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 19HDA

20. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 20HDA Time-Tag Data File Structure

21. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 21HDA ACCUM / FPsplit = no Data File Structure

22. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 22HDA ACCUM / FPsplit = yes or REPEATOBS=N Data File Structure

23. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 23HDA File Naming   TIME-TAG “FP-SPLIT=NO" exposure:     Wavecal exposure science exposure   (401s) (402s)     association file: 4010_asn     wavecal exposure files: 401s_rtg   401s_ctg   401s_flt   401s_x1d     science exposure files:402s_rtg   402s_ctg   402s_flt     association science product file:4010_x1d

24. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 24HDA File Naming   TIME-TAG FP-SPLIT=YES exposure:     Wavecal exposure science exposure   (401s) (402s)   science exposure science exposure   (403s) (404s)   science exposure   (405s)     association file: 4010_asn     wavecal exposure files: 401s_rtg 401s_ctg 401s_flt 401s_x1d     science exposure 1 files: 402s_rtg 402s_ctg 402s_flt 402s_x1d   science exposure 2 files: 403s_rtg 403s_ctg 403s_flt 403s_x1d   science exposure 3 files: 404s_rtg 404s_ctg 404s_flt 404s_x1d   science exposure 4 files: 405s_rtg 405s_ctg 405s_flt 405s_x1d   association science product file: 4010_x1d

25. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 25HDA COS Typical File Sizes

26. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline PDR File Structure Appendix Charles D. (Tony) Keyes 7 December 2000

27. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 27HDA COS Spectroscopic Mode Data Files   TIME-TAG produces each of the following for each FP-SPLIT=NO science and each wavecal exposure: – – a.) raw photon list (_rtg) – – b.) corrected photon list (_ctg) – – c.) “observed” count-rate detector image [not retained] – – d.) corrected effective-count detector image (_flt) – – e.) 1-d extracted spectrum (_x1d)

28. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 28HDA COS Spectroscopic Mode Data Files   TIME-TAG FP-SPLIT=YES exposures: each of the 4 individual component science exposures and all wavecals produce: – – a.) raw photon list (_rtg) – – b.) corrected photon list (_ctg) – – c.) “observed” count-rate detector image [not retained] – – d.) corrected effective-count detector image (_flt) – – e.) 1-d extracted spectrum (_x1d)    and the following is also produced  – – f.) 1-d extracted spectrum summation of _x1d for all 4 individual exposures (_x1s)   TIME-TAG and REPEATOBS combination is not allowed.

29. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 29HDA COS Spectroscopic Mode Data Files   ACCUM produces each of the following for each FP-SPLIT=NO science exposure: – – a.) raw detector image (_raw) – – b.) raw stim-pulse images - one for each stim-pulse (_rs1, _rs2) – – c.) pulse-height distribution [FUV only] (_pha) – – d.) corrected effective-count detector image (_flt) – – e.) "observed" count-rate detector image [not retained] – – f.) corrected stim-pulse images - one for each stim-pulse (_cs1, _cs2) – – g.) 1-d extracted spectrum (_x1d)

30. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 30HDA COS Spectroscopic Mode Data Files  ACCUM FP-SPLIT=YES exposures: each of the 4 individual component science exposures produces:  ACCUM FP-SPLIT=YES exposures: each of the 4 individual component science exposures produces: – a.) raw detector image (_raw) – b.) raw stim-pulse images - one for each stim-pulse [FUV only] (_rs1, _rs2) – c.) pulse-height distribution [FUV only] (_pha) – d.) corrected effective-count detector image (_flt) – e.) "observed" count-rate detector image [not retained] – f.) corrected stim-pulse images - one for each stim-pulse [FUV only] (_cs1, _cs2) – g.) 1-d extracted spectrum (_x1d) – g.) 1-d extracted spectrum (_x1d)  and the following is also produced – h.) 1-d extracted spectrum summation of _x1d for all 4 individual exposures (_x1s) – h.) 1-d extracted spectrum summation of _x1d for all 4 individual exposures (_x1s)  ACCUM FP-SPLIT and REPEATOBS combination is not allowed.  ACCUM FP-SPLIT and REPEATOBS combination is not allowed.

31. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 31HDA COS Spectroscopic Mode Data Files  ACCUM REPEATOBS=N FP-SPLIT=NO exposures: each of the N individual component science exposures produces the following files:  ACCUM REPEATOBS=N FP-SPLIT=NO exposures: each of the N individual component science exposures produces the following files: – – a.) raw detector image (_raw) – – b.) raw stim-pulse images - one for each stim-pulse (_rs1, _rs2) – – c.) pulse-height distribution [FUV only] (_pha) – – d.) corrected effective-count detector image (_flt) – – e.) "observed" count-rate detector image [not retained] – – f.) corrected stim-pulse images - one for each stim-pulse (_cs1, _cs2) – – g.) 1-d extracted spectrum (_x1d)   and the following are also produced – – h.) 1-d extracted spectrum summation of _x1d for all individual exposures (_x1s) – – i.) summation of all individual-exposure corrected effective-count detector images (_fls)   ACCUM FP-SPLIT=YES and REPEATOBS combination is not allowed.

32. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 32HDA Typical COS Output Products by Observation Type*  Observation Uncalibrated Files Calibrated   ========================================================  ACQ, ACQ/PEAKUP rtg, _pha none  --------------------------------------------------------  IMAGING (NUV only) (ACCUM) raw, _pha, _asn _flt, _fls  (repeatobs)  IMAGING (NUV only) (ACCUM) raw, _pha, _asn _flt  (single exposure)  IMAGING (NUV only) (TIME-TAG) _rtg, _pha, _asn _ctg, _flt  (single exposure)  --------------------------------------------------------   * there are NO REPEATOBS for TIME-TAG

33. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 33HDA Typical COS Output Products by Observation Type*  Observation Uncalibrated Files Calibrated Files  =========================================================  SPECTROSCOPY (TIME-TAG) rtg, _asn _ctg, _flt,  (FP-SPLIT) _pha _x1d, _x1s   SPECTROSCOPY (TIME-TAG) _rtg, _asn _ctg, _flt, _x1d  (single) _pha  ---------------------------------------------------------  SPECTROSCOPY (ACCUM) _raw, _pha, _asn _flt,_fls, _x1d,  (FP-SPLIT or repeatobs) rs1, _rs2 _x1s, _cs1, _cs2   SPECTROSCOPY (ACCUM) raw, _pha, _asn _flt, _x1d,_x1s,  (single or repeatobs) _rs1, _rs2, _cs1, _cs2   ---------------------------------------------------------   * there are NO REPEATOBS for TIME-TAG

34. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline PDR Data Volume Overview Charles D. (Tony) Keyes 7 December 2000

35. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 35HDA Reference File Volume  Estimates for COS OPUS reference file requirements based upon AV-03 – assume that L-flats and P-flats are required for each grating – sensitivity curves are required for each wavelength setting of each grating  For files that are applicable at any one time ~1.3 Gbytes are required.  As updated file versions will be delivered periodically, the total volume of reference files will increase ~2.5 Gbytes per cycle.

36. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 36HDA COS Reference File Volume

37. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 37HDA HST and COS Expected Use   At present no combined DRM for operation of all SI in Cycle 12 – – COS usage estimates from COS ISR 99-01, “COS Design Reference Mission and Ground System Volume Requirements” – – compare with anticipated Cycle 11 data volume estimates for ACS/STIS/NICMOS/WFPC2 environment from ACS ISR 97-01 > > estimates, including parallels, for “other SIs” in Cycle 11 – – COS will be used only as prime SI - no COS parallels – – assume COS usage fraction is 25% of available orbits ==> 1000 orbits > > typical COS target is faint > > 6-orbit (maximum duration) visits will be norm > > ~150 visits per year; one 6-orbit visit every other day

38. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 38HDA COS Data Volume Assumptions   For equivalent observing modes, COS/FUV always produces more volume than COS/NUV – – for limiting cases of data volume evaluation, we assume only COS/FUV   TIME-TAG estimates require target brightness assumption – – I. assume “typical” source produces one full onboard 18 Mbyte buffer of events per 3000 sec target visibility period > > corresponds to 4.7E6 photons > > at peak COS S : F ~3.E-14 for FUV; 1.2 E-13 for NUV > > S/N~30 per resel for FUV; ~60 per resel for NUV > > assume wavecal also produces one full buffer > > note: actual source brightnesses of 0.01-0.5E-14 will be common

39. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 39HDA COS Data Volume Assumptions (cont.)  Assumptions for TIME-TAG evaluation (continued): – II. “stressed” maximum volume scenarios: readout at maximum no-loss TIME-TAG rate for designated periods (e.g., for 10-orbit standard visibility [or equivalent 6-orbit CVZ], and extreme 10-orbit CVZ) > at peak COS S : F ~5.E-13 for FUV; 2.0 E-12 for NUV !! – assume all exposures in a visit arrive in OPUS in one day

40. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 40HDA COS Data Volume Summary

41. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 41HDA COS OPUS Data Volume Summary   COS downlink OPUS input requirements – – typical: 0.22 Gbytes/day – – stressed: 1.6 Gb/day (2.7-2.9 Gb/day 10-orbit or 6-orbit CVZ) – – extreme: 5. Gb/day (10-orbit CVZ)   Projected cycle 11 downlink input to OPUS (all SI) – – mean: 2 Gb/day – – peak: 3 Gb/day   Typical COS program produces 0.22 Gb/day downlink to OPUS   COS OPUS online requirements – – typical day and program: 5 Gbytes/day – – stressed: 17 Gb/day (30 Gb/day 10-orbit or 6-orbit CVZ) – – extreme: 50 Gb/day (10-orbit CVZ)

42. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 42HDA COS HDA-input Data Volume Summary   COS OPUS output to HDA (POD plus “raw”) – – typical: 0.76 Gbytes/day – – stressed: 5.7 Gb/day (10 Gb/day 10-orbit or 6-orbit CVZ) – – extreme: 17.5 Gb/day (10-orbit CVZ)   Projected cycle 11 OPUS output to HDA (all SI) – – mean: 9. Gb/day – – peak: 10-11 Gb/day   Typical COS program produces 0.76 Gb/day output to HDA

43. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 43 Preliminary Design Review for COS: COS Associations Ed Hopkins

44. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 44HDA Overview  Basic properties of COS associations – From a packaging point of view – Described in the language the archive uses  The one new feature: from the archive point of view  Comparison to NICMOS associations

45. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 45HDA The One New Feature: for the archive  The archive will have to deal with multiple instances of the same wavecal being ingested with different associations – With STIS the same wavecal was in multiple associations, but it had a different root name. – Because of the COS NICMOS-like packaging, the same wavecal in different associations will have the same root name. – They cannot be differentiated in the archive by their root name > Probably will use association name to distinguish them

46. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 46HDA The Basic Properties of COS Associations  Definitions – Exposure: a set of files connected to one science exposure > With the same root name – Product: a set of files not connected to a science exposure > With the same root name > The output of CALCOS-could be multiple files with the same root  Every association has one and only one product  Every association will have one to N science exposures – N = 4 for Fpsplit – N  for RPTobs  Every association will have one to M wavecals – Where M should be around nine—but is not known  The same wavecal can be in multiple associations

47. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 47HDA NICMOS Associations--the picture

48. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 48HDA COS Associations: the picture

49. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 49HDA COS Associations: the Text  Products mapped to one or N science exposures and 1 to M wavecals.  Mapping is used for – StarView Screens > which exposure level information appears with which product

50. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 50HDA Conclusions  COS associations are a special case of NICMOS associations – With no background targets—only a single product – With duplicate wavecals in different associations  Requirements need to be developed for non-nominal associations

51. SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline Calibration Outline of CALCOS Processing Steps and Keywords Reference Files Associations Reuse of CALSTIS Code Retain CALSTIS nomenclature and file naming conventions to the extent that it is useful to do so.

52. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 52HDA Outline of CALCOS Calibrate each wavecal file:Calibrate each wavecal file: Apply the same calibration as for science data Apply the same calibration as for science data Extract 1-D spectrum Extract 1-D spectrum Determine offset in dispersion direction Determine offset in dispersion direction If data are in TIMETAG mode:If data are in TIMETAG mode: If FUV, determine locations and count rates of stim pins If FUV, determine locations and count rates of stim pins Create corrected timetag table Create corrected timetag table Create accum image from timetag table Create accum image from timetag table Process ACCUM data:Process ACCUM data: Basic 2-D calibration (unless already done) Basic 2-D calibration (unless already done) Extract 1-D spectrum Extract 1-D spectrum Apply wavelength zero point offset from wavecal Apply wavelength zero point offset from wavecal

53. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 53HDA FUV Reduction Process Input Files Processing Step Keyword Switch Output File _rtg or _raw DQITAB Assign data quality DQICORR _ctg or _flt BRFTAB Thermal distortion correction TEMPCORR _ctg or _flt GEOTAB Geometric (INL) correction GEOCORR _ctg or _flt Livetime correction LIVECORR _ctg or _flt PFLTFILELFLTFILE Divide by flat field FLATCORR _ctg or _flt

54. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 54HDA FUV Reduction Process (cont.) Input Files Processing Step Keyword Switch Output File Orbital Doppler correction DOPPCORR _ctg or _flt Heliocentric Doppler correction HELCORR _ctg or _flt PHATAB Filter by pulse height (or verify good pulse height) PHACORR _ctg or _flt BADTTAB Filter by time BADTCORR_ctg _ctg Bin timetag into 2-D accum image BINCORR_flt

55. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 55HDA FUV Reduction Process (cont.) Input Files Processing Step Keyword Switch Output File _fltXTRACTAB Extract 1-D spectrum X1DCORR_x1d _flt Subtract background BACKCORR_x1d LAMPTAB Correct wavelength zero point WAVECORR_x1d _x1dDISPTAB Assign wavelengths DISPCORR_x1d _x1dPHOTTAB Convert count rate to flux FLUXCORR_x1d _x1d Combine FP-split data FPCORR_x1s

56. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 56HDA NUV Reduction Process Input Files Processing Step Keyword Switch Output File _rtg or _raw DQITAB Assign data quality DQICORR _ctg or _flt Livetime correction LIVECORR _ctg or _flt PFLTFILELFLTFILE Divide by flat field FLATCORR _ctg or _flt Orbital Doppler correction DOPPCORR _ctg or _flt Heliocentric Doppler correction HELCORR _ctg or _flt

57. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 57HDA NUV Reduction Process (cont.) Input Files Processing Step BADTTAB Filter by time BADTCORR_ctg _ctg Bin timetag into 2-D accum image BINCORR_flt _fltXTRACTAB Extract 1-D spectrum X1DCORR_x1d _flt Subtract background BACKCORR_x1d LAMPTAB Correct wavelength zero point WAVECORR_x1d _x1dDISPTAB Assign wavelengths DISPCORR_x1d _x1dPHOTTAB Convert count rate to flux FLUXCORR_x1d

58. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 58HDA Reference Images Table Name SuffixDescription Selection Criteria PFLTFILEpfl Pixel-to-pixel flat field DETECTOR LFLTFILElfl Low-order flat field DETECTOR

59. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 59HDA Reference Tables Table name SuffixDescription Selection keywords Selection columns Parameter columns DQITABbpx Locations of bad pixels DETECTORSEGMENT lx, ly, dx, dy, q, type BADTTABtim Bad time intervals start, stop BRFTABbrf Baseline reference frame parameters DETECTORSEGMENT sx1, sy1, sx2, sy2 PHATABpha Pulse height thresholds DETECTORSEGMENT llt, ult

60. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 60HDA Reference Tables (cont.) Table name SuffixDescription Selection keywords Selection columns Parameter columns GEOTABgeo Geometric (INL) correction DETECTORSEGMENT xnelem, xdispl*, ynelem, ydispl* XTRACTAB1dx Locations of spectra and background regions DETECTOR SEGMENT, OPT_ELEM, CENWAVE slope, b_spec, (or b_a, b_b, b_c), bkg1, bkg2, height, bheight, bwidth

61. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 61HDA Reference Tables (cont.) Table name SuffixDescription Selection keywords Selection columns Parameter columns LAMPTABlmp Template lamp spectrum nelem, wavelength*, flux* DISPTABdsp Dispersion coefficients DETECTOR SEGMENT, OPT_ELEM, CENWAVE ncoeff,coeff* PHOTTABpht Photometric throughput DETECTOR SEGMENT, APERTURE, OPT_ELEM, CENWAVE nelem, wavelength*, throughput*

62. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 62HDA Associations CALCOS uses the association table to identify files that will be used in calibrationCALCOS uses the association table to identify files that will be used in calibration Contents of an association:Contents of an association: science data science data wavecal data wavecal data Science data:Science data: timetag timetag accum: accum: spectrum spectrum stim-pulse data (FUV) stim-pulse data (FUV) pulse-height data (FUV) pulse-height data (FUV) both FUV segments in the same file both FUV segments in the same file FP-split or repeatobs in separate files FP-split or repeatobs in separate files

63. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 63HDA Reuse of CALSTIS Code Algorithm reuse, more than code reuse:Algorithm reuse, more than code reuse: Estimate of code reuse: 5%Estimate of code reuse: 5% Estimate of algorithm reuse: 30%Estimate of algorithm reuse: 30% Differences that affect the code but not necessarily the algorithm:Differences that affect the code but not necessarily the algorithm: Wavecal observations will be handled differentlyWavecal observations will be handled differently Different zero point for on-board Doppler shift (affects convolution of flat fields)Different zero point for on-board Doppler shift (affects convolution of flat fields) NUV spectra arranged differently on detectorNUV spectra arranged differently on detector

64. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 64HDA Science Data Processing www.stsci.edu/software/OPUS/ www.dpt.stsci.edu/

65. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 65HDA General OPUS Requirements for Science Data Processing  OPUS will develop the COS pipeline based on the existing pipeline model including OTFR.  Level 0 data packetized and Reed-Solomon corrected by PACOR at GSFC  Receive science telemetry (level 1a data) at STScI as science instrument specific ‘pod files’ – Engineering snapshot included – No on-board compression for COS data  STScI processing on Compaq ALPHA/Tru64 UNIX platform

66. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 66HDA General OPUS Requirements for Science Data Processing (cont.)  OPUS must account for all scheduled exposures.  Convert telemetry to FITS format – Structure tables or data array – Populate header keywords > Keywords to provide metadata for archive catalog  Associate groups of exposures that must process further as a single unit  Execute calibration tasks in pipeline mode  Pass level 1b science data (pod files and uncalibrated science datasets) and jitter files to Hubble Data Archive

67. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 67HDA OPUS Requirements for Thermal Vac  OPUS will develop a COS science data processing pipeline capable of supporting Thermal Vac testing  No database access in Thermal Vac – No support schedule available in PMDB format – Necessary support schedule information to be read in from ASCII file  No associations  No calibration  Data will be archived to HDA  Processing on Sun / Solaris UNIX platform – Need to know which version of Solaris will be used?

68. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 68HDA OPUS Requirements for Thermal Vac (cont.) Thermal Vac OPUS pipeline delivery schedule – Earliest possible COS Thermal Vac currently scheduled for December 10, 2001 – OPUS Thermal Vac pipeline due about two months prior to Thermal Vac, September 24, 2000. – Beta version of OPUS pipeline due about five months prior to Thermal Vac, June 25, 2000.

69. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 69HDA OPUS science data processing pipeline for COS

70. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 70HDA OPUS Processes  Data Partitioning – segments the telemetry stream into standard EDT dataset – fill data inserted if telemetry drop-outs exist > event lost in Time-Tag mode > constructs a data quality image in ACCUM mode to ensure the subsequent science processing does not interpret fill data as valid science data  Support Schedule – gathers proposal information from PMDB – test proposals required for development > test version of PMDB must be populated by TRANS > Thermal Vac support schedule to be input from ASCII file

71. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 71HDA OPUS Processes (cont.)  Data Validation – decodes the exposure and engineering parameters in the telemetry and compares them to the planned values – internal header specification (from Ball) > PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 – flags and indicators need to be determined by the Instrument Scientists  World Coordinate System – implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing – aperture positions must be defined

72. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 72HDA OPUS Processes (cont.)  Generic Conversion – Generic Conversion outputs uncalibrated data > rtg (Time-Tag); raw, rs1, rs2 (ACCUM); spt; trl – data will be output in standard FITS format with image extensions – primary header will contain keywords inherited by all extensions and a null data array – Image extensions > Time-Tag mode will be FITS binary tables > ACCUM mode will be images grouped by imsets consisting of science array and data quality array FUV data will contain separate imsets for each detector (a and b)FUV data will contain separate imsets for each detector (a and b) data quality array will be null if no telemetry dropoutsdata quality array will be null if no telemetry dropouts –calibration generates full data quality array with all other DQ flags

73. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 73HDA OPUS Processes (cont.)  Generic Conversion (cont.) – Required for development > DM-06 to develop algorithms for data formatting > keyword definitions (ICD-19) must be provided by the Instrument Scientists world coordinate definitionsworld coordinate definitions exposure time calculationsexposure time calculations calibration switches and selection criteriacalibration switches and selection criteria calibration file name keywordscalibration file name keywords

74. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 74HDA Keyword specification – keyword name – default value – possible values – units – datatype – short comment for header – long description – header position – DADS table – keyword source The following information must be provided by STScI Science Instrument team for all COS specific keywords using a standard form for keyword database input.

75. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 75HDA Science Instrument Configuration Keywords  Keyword: some possible values – OBSTYPE: Spectroscopy / Imaging – OBSMODE: ACCUM / TIME-TAG – EXPTYPE: ACQ, PEAKUP/XDISP, PEAKUP/DISP, SCI, WAVE, FLAT, DARK, PHA – DETECTOR: FUV, NUV – OPT_ELEM: G130M, G160M, G185M, G225M, G285M, G140L, G230L, MIRROR, MIRRORB – CENWAVE: 1298, 1309, 1320, etc. – APERTURE : PSA, BOA, WCA, FCA (Indicates a COS unique keyword or value)

76. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 76HDA OPUS Processes (cont.)  Data Collector – association table contains all information about product dataset > dataset self-documenting – only associations required for data processing with be constructed in the OPUS pipeline > FP splits > Repeat obs – OPUS will ensure all component exposures are present before processing further > association time-out rules need to be defined by Instrument Scientists > rules for processing incomplete associations need to be defined by Instrument Scientists

77. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 77HDA Wavecal considerations  Current plans for executing wavecals – Wavecal executed after each grating, central wavelength, mirror, or aperture change – Wavecal executed at least once each orbit  Wavecal exposures will be shared between associations – A single wavecal may go into multiple products > For example, wavecal at start of orbit can be shared by all observations in that orbit if it is applicable – Adds complexity to science data processing pipeline

78. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 78HDA Calibration  OPUS will use STSDAS calibration software  run on ALPHA / Tru64 UNIX platform in operations  expands size of dataset – converts integer raw data to real – possible expansion of data quality array for ACCUM mode  Need calibration reference files for testing (at least dummies)

79. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 79HDA Other Science Data Modes  requirements for data content of each of these other science data modes must be defined by Instrument Scientists – target acquisitions – microprocessor memory dump – engineering diagnostic data – cumulative detector image > MAMA cumulative image like STIS for NUV > Possible cumulative image for FUV detector

80. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 80HDA Engineering data processing  Receive engineering telemetry data from CCS at GSFC  Process Engineering data through FGS Data Pipeline  Generate data products to characterize jitter and pointing control information in support of science observations  COS jitter file association packaging will mimic science data associations  No other COS specific requirements

81. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 81HDA Archive Interface  OPUS will present to the archive: – Original data received from PACOR (binary pod files) – Possibly pod file data packaged by observation in FITS format – Output of Generic Conversion (uncalibrated science dataset) in FITS format – Jitter files in FITS format from the engineering telemetry – Data from other science modes (target acquisition, memory dump, engineering diagnostic data, cumulative MAMA image) in FITS format

82. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 82HDA OTFR  In OTFR, data retrieved from the archive are reprocessed from the pod file – Provides HST data user with optimal product at time of retrieval > Calibration updates, bug fixes, and new software features and algorithms available to archive users – OTFR pipeline uses the exact same code as current initial science data processing > Reduces software development and maintenance costs > No science instrument specific code developed for OTFR beyond what is necessary for initial data processing – Adds negligible time for retrievals

83. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 83HDA OTFR

84. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 84HDA CODE REUSE  Core OPUS system (OPUS 12.1) – ~236,000 lines of code – 100% reuse  COS specific processes – Based on FUSE study (Rose et al. 1998, “OPUS: The FUSE Data Pipeline”, www.stsci.edu/software/OPUS/kona2.html) www.stsci.edu/software/OPUS/kona2.html > 5076 lines of code > 71% reuse of existing OPUS modules  Expect > 99% reuse of existing data processing software for COS, based on lines of code. – All SI complexity contained in relatively few lines of code.  Efficient use of existing system!

85. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 85HDA Operational Considerations  No throughput issues anticipated based on COS data processing requirements  OPUS sizing based on SSR limitations in ACS era – COS data volumes should be easily accommodated Hence,  OPUS pipeline throughput for COS could be handled with existing operational environment  No new system requirements for running CALCOS in OPUS  No additional hardware specifications

86. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 86HDA Major Science Data Processing Requirements Summary  Internal header specification – DM-06 to document content and format of science internal header – PDB (EUDL.DAT, TDFD.DAT) defined and fully populated  Keyword definitions  Flags and indicators for Data Validation  Aperture definitions

87. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 87HDA Test Data Requirements  Test data from detectors on optical bench expected in March 2001 and from integrated instrument in August 2001  Test data to be provided by IDT/Instrument Scientists and Engineers should include all science modes  Test data must include – PMDB population and PDB definition – spectra that enable OPUS to determine proper orientation – list of possible error conditions to simulate – data that simulate error conditions – enough data for throughput test – engineering data to test jitter file production

88. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 88 HUBBLE DATA ARCHIVE Warren King 7 December, 2000

89. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 89HDA Hubble Data Archive COS Pipeline PDR  Will cover two main areas of the HDA – Ingest – Distribution

90. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 90HDA Ingest Issues  Associations – No new types of associations  Keywords and headers design – AT, DPT, SSG and IDT have been working together to minimize impact on current systems  No change to OPUS/DADS interface  95% Code Reuse

91. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 91HDA Ingest Issues (cont.)  Data rate can be supported  Data to be Archived > No intermediate files > FITS compression plus gzip file compression > On The Fly Reprocessing

92. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 92HDA Distribution Issues  New StarView screens  Minor changes to: – Add COS data to PI paper products – Add COS to HST remote archive site distribution – Define default file types for CAL, UNCAL, and DQ flags on StarView retrieves – 98% code reuse

93. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 93HDA Distribution Issues (cont.)  COS Dataset Size – Reduce data being sent to users > Gzip files Compressed FITS format will reduce outbound network loadCompressed FITS format will reduce outbound network load – Alternative media if Internet becomes the bottleneck > Tape (current) > CD (future) > DVD (future)

94. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 94HDA Peak Historical Ingest Rates  All Data (since August ’00) – 1 day ingest rate over 16GB – 3 days 10GB - 16GB – 4 days 8 GB – 10 GB – 6 month average of 3GB/day

95. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 95HDA Archive Statistics

96. Archive traffic for the past year, in Gbytes (Figures in parentheses are Gbytes/day) Year Month Archived Retrieved ---- ----------- --------------- -------------- 1999 May 96.4 ( 3.11) 385.6 (12.44) June 109.1 ( 3.64) 413.6 (13.79) July 148.9 ( 4.80) 585.7 (18.89) August 105.8 ( 3.41) 512.0 (16.52) September 114.6 ( 3.82) 490.5 (16.35) October 102.2 ( 3.30) 391.7 (12.64) November 44.9 ( 1.50) 331.5 (11.05) December 14.8 ( 0.48) 300.6 ( 9.70) 2000 January 80.6 ( 2.60) 370.5 (11.95) February 106.3 ( 3.67) 441.0 (15.21) March 102.1 ( 3.29) 464.7 (14.99) April 99.8 ( 3.33) 378.2 (12.61) May 99.4 ( 3.21) 332.5 (10.73) June 95.2 ( 3.17) 440.2 (14.67) July 108.9 ( 3.51) 394.1 (12.71) August 95.2 ( 3.07) 637.5 (20.56) September 102.0 ( 3.40) 557.8 (18.59) October 99.8 ( 3.22) 515.4 (16.63)

97. StarView 6

98. Archive traffic for the past year, in Gbytes (Figures in parentheses are Gbytes/day) Year Month Archived Retrieved ---- ----------- --------------- -------------- 1999 May 96.4 ( 3.11) 385.6 (12.44) June 109.1 ( 3.64) 413.6 (13.79) July 148.9 ( 4.80) 585.7 (18.89) August 105.8 ( 3.41) 512.0 (16.52) September 114.6 ( 3.82) 490.5 (16.35) October 102.2 ( 3.30) 391.7 (12.64) November 44.9 ( 1.50) 331.5 (11.05) December 14.8 ( 0.48) 300.6 ( 9.70) 2000 January 80.6 ( 2.60) 370.5 (11.95) February 106.3 ( 3.67) 441.0 (15.21) March 102.1 ( 3.29) 464.7 (14.99) April 99.8 ( 3.33) 378.2 (12.61) May 99.4 ( 3.21) 332.5 (10.73) June 95.2 ( 3.17) 440.2 (14.67) July 108.9 ( 3.51) 394.1 (12.71) August 95.2 ( 3.07) 637.5 (20.56) September 102.0 ( 3.40) 557.8 (18.59) October 99.8 ( 3.22) 515.4 (16.63)

99. COS Pipeline and Archives Schedule

101. COS PIPELINE & ARCHIVES PDR 7 December 2000 Space Telescope Science Institute 101HDA Resource Estimates