6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1

6e-2 Intro/Overview Science Data Product design shows what the DMS software should produce. Software design then shows how to generate data products. JWST science data products generated by DMS are dynamic due to updates to calibration reference data and algorithms, increased knowledge of the science instruments learned in the early stages of the mission, and DMS software error corrections. Intent of DMS Science Data Product Generation is to provide data users with basic reduced data products they need to begin detailed analysis. DMS data products conform to Flexible Image Transport System (FITS) convention. standard archival format for astronomical data S&OC System Design Review #1

6e-3 DMS Data Flow – Data Product Generation S&OC System Design Review #1

6e-4 JWST Up-the-Ramp Readout S&OC System Design Review #1

6e-5 Data Level Definition S&OC System Design Review #1

6e-6 Science Data Product Inputs S&OC System Design Review #1

6e-7 Generation of Original Exposure Data Files S&OC System Design Review #1

6e-8 Level 1 FITS File Structure S&OC System Design Review #1

6e-9 Science Data Cube single exposure, single detector S&OC System Design Review #1

6e-10 Image Orientation Detector and science image frame orientations as seen looking through the optics toward the sky S&OC System Design Review #1 red – detector frame blue – science frame green – apertures NIRCam NIRSpec MIRI NIRISS FGS1FGS2 Transformation requires rotation, parity flip, and/or shift.

6e-11 Information Flow to Level 1 FITS Headers S&OC System Design Review #1

6e-12 Level 1a-Level 1b Header Keyword Population S&OC System Design Review #1

6e-13 Files in an Exposure Data Set File typeData level created Science Instrument Format uncalibrated1ballFITS pixel data image and group information ASCII table MSA metadata1bNIRSpecFITS ASCII table reference stars1bNIRSpecFITS ASCII table jitter file1ballFITS ASCII table count rate image2aallFITS image calibrated image2ballFITS image processing history1a-2balltext S&OC System Design Review #1

6e-14 Exposure File Naming Convention At data levels 1 and 2, the science data files are FITS format files that contain the pixel values for a single exposure from a single detector. Goals for file name: DMS data user able to map the data file to the observation input through APT. Operations staff can map every exposure to a construct in the JWST commanding scripts. The file name is based on the following exposure hierarchy: Program (PPPPP) Observation number (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure number (EEEEE) Example: jw _ _ _ _.fits jw _01101_00012_nrcalong_uncal.fits Visit ID - - explicitly embedded in exposure file name Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each exposure within a single visit. S&OC System Design Review #1

6e-15 Guide Star Pipeline Guide Star Pipeline detailed design to be addressed in DMS SDR4 Jitter data file generated by Guide Star Pipeline Separate process from science data pipelines Data product created at visit level Guide Star (GS) acquisition images About GS - Full frame image, Acq 1: 128x128, Acq 2: 32x32 At science attitude: Acq 1: 128x128, Acq 2: 32x32 Fine guide box: 16Hz (64 msec.) May not want to save all 8x8 fine guide data Centroids in engineering data for jitter Include additional pointing information in engineering data S&OC System Design Review #1

6e-16 NIRSpec MSA Data Flow S&OC System Design Review #1

6e-17 NIRSpec MSA Metadata File Structure S&OC System Design Review #1 Data array for each shutter planned open or closed Slitlet information, background shutter, fraction of source in shutter Source id, source name, alias, RA, Dec

6e-18 Generation of Calibrated Exposure Data Files S&OC System Design Review #1

6e-19 Level 2 FITS File Format S&OC System Design Review #1

6e-20 Level 3 Data Products - Associations Relationships between multiple exposures are captured in an association. An association is a means of identifying a set of exposures that belong together and may be dependent upon one another. The association concept permits exposures to be calibrated, archived, retrieved, and reprocessed as a set rather than as individual objects. For initial processing, an association and its level 3 data products will not extend beyond a single observation. Design will include the capability to support multi-observation and multi-program associations. For each association, DMS will generate the most combined and least combined data products. Intermediate data products will not be saved. For imaging associations, the most combined will be either a mosaic or a dither product. S&OC System Design Review #1

6e-21 Association Components S&OC System Design Review #1 Science Instrument Data ProductsRelated FilesContemporaneous Calibration Files NIRCamMosaic combined image (multiple detectors) Dither combined image (if no mosaic) Individual exposures (single detector; levels 1a, 1b, 2a, 2b) Coronagraphic target acquisition Jitter data Flats (taken with science exposure) Persistence darks NIRSpecExtracted 1-d and 2-d spectra over both detectors MSA Source Catalog (level 4) Individual exposures (single detector; levels 1a, 1b, 2a, 2b) Target acquisition images Confirmation images Jitter data MSA Pre-image MSA Metadata file Reference star file Background exposures (may be shared among exposures) Auto-flats Wave-cals MIRIMosaic combined image Dither combined image (if no mosaic) Extracted 1-d and 2-d spectra Extracted 1-d and 2-d spectra over both IFU detectors Individual exposures (single detector; levels 1a, 1b, 2a, 2b) Coronagraphic target acquisition images Spectroscopic target acquisition images Jitter data Imager internal flats MRS internal flats Chopped background image (different visit, no interrupts) Persistence darks NIRISS/FGSMosaic combined image Dither combined image (if no mosaic) Individual exposures (single detector; levels 1a, 1b, 2a, 2b) Jitter data Point source image for PSF characterization Persistence darks

6e-22 Association Data Products Association Pools Created by Science Data Processing Contains a list of all potential exposures that may form an association Association Table Created in calibration pipeline environment from Association Pool information Contains list of exposures to be used to create a single Level 3 Data Product Level 3 Data Products Formed from exposures listed in Association Table S&OC System Design Review #1

6e-23 Association Processing All exposures – even single, unassociated exposures – will have a level 3 data product. Well-established image rectification software (Drizzle) will be run at level 3 on all exposures. All data will get a source based, level 3 file name. S&OC System Design Review #1

6e-24 Level 3 Image FITS File Format S&OC System Design Review #1 Most combined image FITS table of selected keywords from input exposures Additional extensions to be defined by calibration working group

6e-25 Level 3 Data Product File Name Shift in naming convention for level 3 data product file names The association data product name will provide an indication of the file content. Based on data discovery lessons learned from MAST High Level Science Product (HLSP) and Hubble Legacy Archive (HLA) experience, A proposed name for the JWST level 3 data products was patterned after the HLSP data file names. ( The format for the level 3 data product file name would be: jw _ _ _ _.fits is the program identifier is the target name, which may contain proposer defined non- alphanumeric characters and an epoch is nircam, nirspec, miri, niriss, or fgs is the optical element or optical element combination used indicates the type of data product, e.g., image Examples from SODRM: jw94010_m31_nircam_f356w_image.fits jw95020_cdfs-07-e1_nircam_f115w_image.fits jw93060_w51a+g _miri_f1500w_image.fits S&OC System Design Review #1

6e-26 Data Level Format Summary Data levelFormat Level 0CCSDS packets Level 0.5CCSDS packets sorted by exposure and detector Level 1a4-d data cube Level 1b4-d data cube Level 2a3-d data cube – collapse groups within each integration into a count-rate image and Single 2-d image – collapse integrations into one count-rate image Level 2bSingle 2-d image Level 3Single or combined, rectified 2-d image; extracted spectra S&OC System Design Review #1