File FormatsFile Formats File Name ConventionsFile Name Conventions File ContentsFile Contents Association Contents & RulesAssociation Contents & Rules File SizesFile Sizes WFC3 Datasets and Associations

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations2 Specifications & Acknowledgements AV-03AV-03 –Data set content requirements ISR WFC ISR WFC –Detailed description of data formats WFC3 design follows closely ACS & NICMOS specifications, originally developed by:WFC3 design follows closely ACS & NICMOS specifications, originally developed by: –Jedrzejewski, Stiavelli, Clampin –MacKenty, Bushouse, Skinner

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations3 File Format Overview All file format and content assumptions established at PDR: File formats utilize STIS, NICMOS, and ACS heritageFile formats utilize STIS, NICMOS, and ACS heritage All science files are FITS with image and table extensionsAll science files are FITS with image and table extensions Images represented by several data arrays, known as image set (“imset”)Images represented by several data arrays, known as image set (“imset”) 2 UVIS chip images stored as 2 IMSETs in one file (same as ACS WFC)2 UVIS chip images stored as 2 IMSETs in one file (same as ACS WFC) IR MultiAccum readouts stored as multiple IMSETs in one file (same as NICMOSIR MultiAccum readouts stored as multiple IMSETs in one file (same as NICMOS Associated exposures – including members of CR-SPLITs, REPEAT-OBS, and Patterns – stored in separate filesAssociated exposures – including members of CR-SPLITs, REPEAT-OBS, and Patterns – stored in separate files

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations4 Science File Organization IMSET contentsIMSET contents –UVIS (same as STIS and ACS) Science Image (SCI)Science Image (SCI) Error Image (ERR)Error Image (ERR) Data Quality Flags (DQ)Data Quality Flags (DQ) –IR (same as NICMOS) Science Image (SCI)Science Image (SCI) Error Image (ERR)Error Image (ERR) Data Quality Flags (DQ)Data Quality Flags (DQ) Number of Samples (SAMP)Number of Samples (SAMP) Exposure Time (TIME)Exposure Time (TIME) Primary Header SCI Extension ERR Extension DQ Extension x 2 IMSETs Primary Header SCI Extension ERR Extension DQ Extension SAMP Extension TIME Extension x NSAMP IMSETs

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations5 WFC3 File Names WFC3 will retain ACS and some NICMOS name conventions (no new file suffixes) SuffixTypeContents _rawImage Raw science image _asnTable Association table _sptImage Support file _trlTable Trailer file _imaImage IR Intermediate MultiAccum image _fltImage Calibrated image (for UVIS CR-SPLIT & REPEAT-OBS=1 and all IR) _crjImage Combined, CR-rejected image (for CR-SPLIT or REPEAT-OBS > 1)

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations6Associations Associations will be formed for:Associations will be formed for: –Multiple CR-SPLIT exposures (single pointing) –Multiple REPEAT-OBS exposures (single pointing) –Pattern exposures (multiple pointings) –Any and all combinations of the above Associations will be formed only for those pattern exposures that can be executed on a single set of guide starsAssociations will be formed only for those pattern exposures that can be executed on a single set of guide stars Patterns too large for a single set of guide stars will not form an associationPatterns too large for a single set of guide stars will not form an association –associations of “inner” groupings (e.g. CR-SPLITs at pattern positions or small dithers at pattern positions) will be formed TransVerse may be capable of lifting this restriction, if desiredTransVerse may be capable of lifting this restriction, if desired

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations7 Association Products WFC3 associations will produce sub-products & main products:WFC3 associations will produce sub-products & main products: –Sub-products: combined CR-SPLIT imagescombined CR-SPLIT images combined REPEAT-OBS imagescombined REPEAT-OBS images –Main Products: Combined dithered/mosaiced imagesCombined dithered/mosaiced images Inputs can be individual exposures (single exposure per pattern pointing), or sub-products (multiple exposures per pointing)Inputs can be individual exposures (single exposure per pattern pointing), or sub-products (multiple exposures per pointing) Allowed maximum of 19 products in one associationAllowed maximum of 19 products in one association –18 sub-products + 1 main product (current max used is 9) Main product will not always be presentMain product will not always be present –e.g. CR-SPLIT or REPEAT-OBS > 1, but no dither pattern

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations8 Incomplete Associations Given that: Every association must have at least one exposure and one productEvery association must have at least one exposure and one product Every exposure must map to at least one productEvery exposure must map to at least one productTherefore: If at least one exposure is present, a product will be createdIf at least one exposure is present, a product will be created –calwf3 design and processing capability allows for this “Stragglers” are allowed and will be processed and archived as an individual exposure“Stragglers” are allowed and will be processed and archived as an individual exposure

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations9 Sample Association #1 CR-SPLIT=3 No Dither EXP-CRJ PROD-CRJ EXP-CRJ Exposure Sub-Product No Main Product

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations10 Sample Association #2 CR-SPLIT=2 2-position Dither Position #1 Position #2 EXP-CR1 PROD-CR1 EXP-CR2 PROD-CR2 PROD-DTH Exposure Sub-Product Main Product

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations11Keywords UVIS and IR images will use separate sets of detector- and mode-specific keywordsUVIS and IR images will use separate sets of detector- and mode-specific keywords Reuse ACS WFC for WFC3 UVIS and NICMOS for WFC3 IR keywords (as appropriate)Reuse ACS WFC for WFC3 UVIS and NICMOS for WFC3 IR keywords (as appropriate) UVIS and IR mode keywords are self-consistent and consistent with ACS (and predecessor STIS) usageUVIS and IR mode keywords are self-consistent and consistent with ACS (and predecessor STIS) usage Science file keywords (1 st draft) delivered to OPUS May 2001Science file keywords (1 st draft) delivered to OPUS May 2001 –Update to be delivered soon from recent DM-06 update by Ball All internal header ED snapshot entries to be stored as keywords in UDL extensions of SPT filesAll internal header ED snapshot entries to be stored as keywords in UDL extensions of SPT files

WFC3 PIPELINE CDR H. Bushouse October 16, 2001WFC3 Datasets & Associations12 WFC3 File Sizes UVIS IR FileTypeSize ASNTable < 0.1 MB < 0.1 MB SPTImage ~ 0.1 ~ 0.1 TRLTable < 0.1 < 0.1 RAWImage FLT & CRJ Image DRZ (1 image) Image ~ 180 DRZ (max) 1 Image ~ 461 FileTypeSizeASNTable < 0.1 MB < 0.1 MB SPTImage < 1 < 1 TRLTable < 0.1 < 0.1 RAWImage < 33 2 < 33 2 IMAImage < FLT & CRJ Image DRZ (1 image) Image ~ 12 ~ 12 DRZ (max) 1 Image ~ 35 ~ 35 1 Assumes max 100” dither 2 2 MB per readout MB per readout