1. 5-1 DMS Data Products Nov. 6, 2013 S&OC Data Management System Design Review 3

2. 5-2 Presentation Outline Data Definition Review NIRSpec MSA Metadata File Imaging Associations Future Data Product Design Nov. 6, 2013S&OC Data Management System Design Review 3

3. 5-3 Data Product References DMS Requirements Document (JWST-STScI-002249, JWST-RQMT- 002941) Generation of Level 1a Science Data Products (JWST-STScI- 002566) DMS Level 1 & 2 Data Product Design (JWST-STScI-002111) S&OC Subsystem IRCD (JWST-IRCD-013128, JWST-STScI-000905) Contains specification for data exchanged between S&OC subsystems PRDS to Users IRCD Volume III: S&OC Subsystems (JWST-STScI- 000949) Contains specification for S&OC data contained in PRD Definition of the Flexible Image Transport System (FITS), version 3.0, A&A 524, A42 (2010) FITS standard Nov. 6, 2013S&OC Data Management System Design Review 3

4. 5-4 Data Level Definition Nov. 6, 2013S&OC Data Management System Design Review 3

5. 5-5 Exposure File Naming Convention At data levels 1 and 2, the science data files are FITS format files that contain the pixel values for a single exposure from a single detector. Goals for file name: DMS data user is able to map the data file to the observation input through the Astronomer's Proposal Tool (APT). Operations staff can map every exposure to a construct in the JWST commanding scripts that was used to make the observation request. The file name is based on the following exposure hierarchy: Program (PPPPP) Observation (OOO) Visit (VVV) Visit Group (GG) Parallel Sequence id (S, 1 prime, 2-5 parallel) Activity (AA, base 36) Exposure id (EEEEE) Example: jw _ _ _ _.fits jw12345012009_01101_00012_nrcalong_uncal.fits Visit ID explicitly embedded in exposure file name Exposure number resets with each new activity The exposure number within the file name does not necessarily monotonically increase for each exposure within a single visit. Nov. 6, 2013S&OC Data Management System Design Review 3

6. 5-6 NIRSpec MSA Nov. 6, 2013S&OC Data Management System Design Review 3

7. 5-7 Relevant DMS Requirements Nov. 6, 2013S&OC Data Management System Design Review 3 Req. #RequirementSOC parent DMS-2343.3.2.1 Science Data Product Generation The DMS shall generate Science Data Products from Science Data. 3.4.5.1.3 Science Data Product Generation [SOC-529] DMS-1335.1.6.1 Include PPS Information in Header Keyword Values SDP shall populate the FITS data headers with proposal, planning, and scheduling information received from PPS. 3.4.2.1.2 Level 1 Processing [SOC-1689] DMS-1926.2.1.1 Generate Science Data Products The DMS shall generate Calibrated Science Data Products from Science Data for all supported modes of the JWST science instruments, as defined in section 3.1.3 of the S&OC Requirements Document (JWST- RQMT-002032, JWST-STScI-000046) under Science Instrument Capabilities. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-4606.2.1.4 Create Level-2 Data Products The Calibration Software shall create Level 2 Science Data Products as specified in the User Handbooks, OP-07. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-4616.2.1.5 Create Level-3 Data Products The Calibration Software shall create Level 3 Science Data Products as specified in the User Handbooks, OP-07. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-4806.2.2.17 Spectra Combination The Calibration software shall have the capability of combining spectra extracted from different exposures. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-4816.2.2.18 Background Subtraction The Calibration software shall have the capability of subtracting the background from a spectrum. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-4826.2.2.19 Spectroscopic Extraction The Calibration software shall have the capability of extracting a 1-d spectrum from 2-d spectroscopic data. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-5496.3.1.1 Association Definition The DMS shall define Associations that consist of exposures that may be combined during calibration. 3.4.3.1.2 Science Data Calibration [SOC-513] DMS-5536.3.1.5 Contemporaneous Calibration Associations The DMS shall associate contemporaneous calibration exposures with the relevant science exposures. 3.4.3.1.2 Science Data Calibration [SOC-513]

8. 5-8 NIRSpec MSA Reference Documentation “NIRSpec Data Associations Study: Associating MSA Spectral Data in the Calibration Pipeline and MSA Archive File Structures,” T Beck, J Valenti, J Muzerolle, T Keyes, JWST-STScI-003366, May 24, 2013 Association product study led to definition of data products “Further Definition of the NIRSpec Science Templates,” D. Karakla, T. Beck, W. Blair, C. Keyes, J. Muzerolle, K. Pontoppidan, D. Soderblom, J. Tumlinson, and J. Valenti, JWST-STScI-002288, 18 March 2011. “STScI NIRSpec Calibration Pipeline Processing Description – Rev. A,” Tracy Beck, JWST-STScI-001859, 15 June 2011. NIRSpec MSA Planning Tool “Optimization Strategies for the NIRSpec MSA Planning Tool,” James Muzerolle, JWST-STScI-001851, 9 December 2009. “MSA Planning Tool – A First Algorithm to Plan an Observation,” Diane Karakla and the MSA Tool Algorithm Working Group, JWST-STScI- 003003, 22 October 2012. “MSA Planning Tool Design Requirements,” Diane Karakla and the MSA Tool team, JWST-STScI-002971, 4 December 2012. Nov. 6, 2013S&OC Data Management System Design Review 3

9. 5-9 NIRSpec MSA Design Summary Data files Level 1 and 2 science data files based on exposure/detector Level 3 science data files organized by sources DMS metadata file captures MSA metadata from PPS database Archive paradigm change Save only least combined and most combined data for a program Least combined data – individual exposure data products at level 1 and 2 - No pixel resampling Most combined data – level 3 data with the most sources - Changes over time as more visits execute - New combined products should always be generated using the exposure-level, least combined products as the inputs - New combined products replace previous versions in the archive Nov. 6, 2013S&OC Data Management System Design Review 3

10. 5-10 MSA Data Flow Nov. 6, 2013S&OC Data Management System Design Review 3

11. 5-11 NIRSpec MSA Glossary (1/2) Pre-image – high quality image provided by PI in FITS format used to select sources for MSA observations relative accuracy of target coordinates from the pre-image must be known to within 5 mas Target – position to point the telescope (V2,V3) with fixed orientation Reference stars – set of 8-20 reference stars to provide relative astrometry for target acquisition Catalog – set of sources used to plan observation Not all sources may be observed Sources – objects of observation in MSA shutter Target set – set of objects observed simultaneously Target set needs to prevent overlapping spectra for any given configuration Nov. 6, 2013S&OC Data Management System Design Review 3

12. 5-12 NIRSpec MSA Glossary (2/2) Slitlet – set of related micro-shutters For example, one shutter contains the source, other shutters in the same column contain local background For medium and high resolution gratings, only one open shutter per row - Prisms will not spread spectra as far as medium and high resolution gratings. In that case, multiple shutters can be open per row without overlapping spectra. Rows with a failed open shutter are blocked from use Rows with a failed closed shutter are tolerated as long as the failed closed shutter is not part of the slitlet Sources can be dithered within the slitlet by small angle maneuvers - Improves accuracy of background subtraction - Dithers the source spectrum on the detector between sub-exposures so that bad detector pixels can be corrected Nod – small dither to move source position within slitlet without MSA reconfiguration Nov. 6, 2013S&OC Data Management System Design Review 3

13. 5-13 NIRSpec MSA – DMS Metadata File Topics to be covered: How DMS/SDP extracts MSA metadata from PPS database Additional MSA information captured in calibration reference files DMS MSA metadata data product design DMS MSA Metadata File Science Data Processing component extracts information from PPS and generates the MSA Metadata File MSA Metadata File used by archive users and calibration software - Archive users do not have access to PPS database - Provides interface to PPS database for calibration software Nov. 6, 2013S&OC Data Management System Design Review 3

14. 5-14 NIRSpec MSA Metadata Information Flow Nov. 6, 2013S&OC Data Management System Design Review 3

15. 5-15 DMS Relevant PPS Tables Nov. 6, 2013S&OC Data Management System Design Review 3

16. 5-16 Comments on PPSDB Tables nirspec_msa_at_pointing Allows link between Observatory Status File and NIRSpec MSA PPS database tables msa_default_state = ‘open’ or ‘closed’ nirspec_msa_config Lists shutters not in default state Configuration id unique within a program nirspec_msa_metadata DMS specific information msa_metadata_id unique within a program Information on shutter level slitlet_id can apply to multiple shutters slitlet_id unique within program shutter_state =‘open’ or ‘closed’: Can force shutters closed to mask bright source background = T or F background shutter can be assigned to multiple sources PPS to compute source PSF position in shutter in x and y directions of NIRSpec ideal coordinate frame in units of fraction of shutter length and width Link to nirspec_sources for source information nirspec_target_acq New visit and target acq required for configuration change Program, observation, visit, and order_number can be used to link to exposure specification 8-20 reference stars per target acq. Link to nirspec_sources for source information nirspec_sources pre_image_id unique within a program Map pointing to pre-image Nov. 6, 2013S&OC Data Management System Design Review 3

17. 5-17 PPS nirspec_msa_metadata Table Details descriptionfielddata typevalues Metadata idmsa_metadata_idI4unique within a program Row for the open shuttershutter_rowI41..365 Column for the shuttershutter_columnI41..171 Quadrant of the shuttershutter_quadrantI21..4 Shutter is opened or closedshutter_stateVC(6) Open/Closed Source associated with this shuttersource_idI4references a source Is this backgroundbackgroundchar(1)Y/N Identifier to indicate the logical group of shutters to which this shutter belongs. slitlet_idI4unique within a program Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame x direction estimated_ source_in _ shutter_x R8fraction of shutter size in x Source PSF position as offset from the shutter center in NIRSpec ideal coordinate frame y direction estimated_ source_in _ shutter_y R8fraction of shutter size in y Nov. 6, 2013S&OC Data Management System Design Review 3

18. 5-18 PPS nirspec_sources Table Nov. 6, 2013S&OC Data Management System Design Review 3 Field nameTypeDescriptionUnits/range programI4Program number1…99999 source_idI4Unique ID of a MSA source within a program aliasVC31Optional name that the observer may give to a source catalog_idVC20PI catalog from which the source was taken raR8Coordinates of source in right ascension degrees decR8Coordinates of source in declinationdegrees pre_image_idVC20Name of the pre-image that was used to measure position for this source

19. 5-19 Additional Files Input to Calibration Shutter stuck open/closed file(s) and MSA to detector map to be delivered to CRDS through NIRSpec Instrument Team as reference files Information not provided in MSA Metadata File File(s) to indicate shutters stuck open or closed Stuck shutters may change between time of phase II proposal prep and visit execution Also provides contrast – fraction of light to get through shutter (even when shutter open or closed) Distortion maps Needed for distortion correction in MSA plane - Not a pixel map, polynomials will be used to characterize distortion - Distortion in detector plane characterized in SIAF Used for on-board target acquisition Needed for DMS calibration Same at any given point in time regardless of aperture The mapping file could change from the time of planning to the time of execution Nov. 6, 2013S&OC Data Management System Design Review 3

20. 5-20 Science Exposure Header Keyword Information Pointer to pre-image file name One pre-image per configuration A pre-image can generate multiple configurations Pointer to stuck open/closed CRDS reference file Pointer to MSA to detector map CRDS reference file Pointer to MSA metadata file Information to map exposure to configuration and metadata Program Observation Visit Pointing sequence number MSA configuration id MSA metadata id MSA default state Engineering parameter needed to provide grating wheel position Nov. 6, 2013S&OC Data Management System Design Review 3

21. 5-21 Exposure to ID Correspondence Each exposure has a single configuration id and a single metadata id Level 1 and 2 files for detectors NRS1 and NRS2 use same configuration id and metadata id A configuration id can be applied to multiple exposures For example, exposures through multiple gratings with same configuration There can be multiple metadata ids within configuration For example, background shutter changes between nods A metadata id can be applied to multiple exposures, but will typically be applied to a single pointing For example, exposures through multiple gratings Nov. 6, 2013S&OC Data Management System Design Review 3

22. 5-22 NIRSpec MSA Metadata Data Product Generated by SDP (Level 1 processing) Includes information from planning database FITS format file One file for each unique MSA configuration Uniqueness defined by program, msa_config_id File name format: jw _ _msa.fits - Visit_id 11 characters: - Msa_config_id: 2 base-36 characters File needs to be associated with all exposures and higher level data products using that configuration Multiple metadata files per data product Metadata data product generated at time of first exposure using that configuration Nov. 6, 2013S&OC Data Management System Design Review 3

23. 5-23 FITS File Structure Nov. 6, 2013S&OC Data Management System Design Review 3

24. 5-24 Shutter State Data Data array to show anticipated state of all shutters Planned shutter state based on PPS database information Calibration can update shutter state information from reference files and confirmation images FITS data array values Closed - 0, Open - slitlet_id If shutter open without being assigned a slitlet_id, set data array value = -1. Imaging mode settings TBD Nov. 6, 2013S&OC Data Management System Design Review 3

25. 5-25 Shutter Info Extension Table listing all shutters included in a slitlet Fields would include: Mimics PPS nirspec_msa_metadata table. Assumes all useful shutters assigned a slitlet_id Background shutters or slitlets can be assigned to multiple sources - Quadrant, row, and column do not necessarily define a unique table row metadata _id shutter_ quadrant shutter _ row shutter_ column slitlet _ id shutter _ state back ground source _ id estimated_ source_in _ shutter_x estimated_ source_in_ shutter_y Nov. 6, 2013S&OC Data Management System Design Review 3

26. 5-26 Source Info Extension Table listing all sources Fields would include: May include additional information from nirspec_sources_flux Source id _ Where nnn is source number within program Source name Source names standardized Format: _ _ _ Source names should be assigned within PPS Need to propagate source name from level 2 extensions to level 3 data files RA and Dec ICRS reference frame and J2000.0 epoch - This information to go in the source extension header keywords. Nov. 6, 2013S&OC Data Management System Design Review 3 source_idsource namealiascatalog_idradec

27. 5-27 Imaging Association Introduction Nov. 6, 2013S&OC Data Management System Design Review 3

28. 5-28 DMS SDR3 Association Presentations What is an association (Swade) How to form an association pool (Swam/Alexov) How to generate associations (Bushouse) Separate association design review in 2014 with more details Nov. 6, 2013S&OC Data Management System Design Review 3

29. 5-29 6.3.1Association Properties Nov. 6, 2013S&OC Data Management System Design Review 3

30. 5-30 6.3.4Associations in the Archive Nov. 6, 2013S&OC Data Management System Design Review 3

31. 5-31 Goals of Design Identify the associated (level 3) data products for each JWST science instrument Develop a naming convention for the associated data products Identify individual files that will be associated with the data products Identify the parameters necessary to generate each association Identify the source of the parameter values Nov. 6, 2013S&OC Data Management System Design Review 3

32. 5-32 Association Design Process Started with designing associations for imaging data products Imaging associations due in DMS Build 3 Considered SODRM programs as use cases Built MIRI and NIRCam imaging association pool and considered data products for SODRM 93060 - Galactic Stellar Clusters Nov. 6, 2013S&OC Data Management System Design Review 3

33. 5-33 Association Components Nov. 6, 2013 S&OC Data Management System Design Review 3

34. 5-34 Associated Data Products Imaging Mosaic combined image NIRCam mosaic contains multiple detectors Dither combined image (if no mosaic) Spectra NIRSpec Extracted 1-d and 2-d spectra over both detectors NIRSpec MSA Source Catalog (level 4) MIRI Extracted 1-d and 2-d spectra MIRI Extracted 1-d and 2-d spectra over both IFU detectors Nov. 6, 2013S&OC Data Management System Design Review 3

35. 5-35 Associated Files Individual Exposures (single detector; levels 1a, 1b, 2a, 2b) Jitter data NIRCam Coronagraphic target acquisition NIRSpec Target acquisition images NIRSpec Confirmation images NIRSpec MSA Pre-image NIRSpec MSA Metadata file NIRSpec Reference star file NIRSpec Background exposures (may be shared among exposures) MIRI Coronagraphic target acquisition images MIRI Spectroscopic target acquisition images Nov. 6, 2013S&OC Data Management System Design Review 3

36. 5-36 Contemporaneous Reference Files NIRCam flats NIRSpec Auto-flats NIRSpec Wave-cals MIRI Imager internal flats MIRI MRS internal flats MIRI Chopped background image MIRI Different visit, no interrupts Persistence darks (in the distant future) Nov. 6, 2013S&OC Data Management System Design Review 3

37. 5-37 Imaging Associations Nov. 6, 2013 S&OC Data Management System Design Review 3

38. 5-38 Template Definition Template – APT input used to define individual Observations. An Observation Template is completed in APT which provides the necessary information for the internal scheduling software to generate the observing sequence (integrations, exposures, and visits) to execute on the telescope. The templates are selected based on an observing strategy (e.g. MIRI imaging, MIRI low resolution spectroscopy) required for observations. Only one instrument template can be used for any Observation. An Observation is a single instance of a template. One-one correspondence exists between an observation and template instance Nov. 6, 2013S&OC Data Management System Design Review 3

39. 5-39 Observation Data Products In general, level 3 data products do not extend beyond one observation. A mechanism is needed for PSS to let DMS know about exceptions: Multi-observation mosaics Shared contemporaneous calibration exposures Latent image correction (Defer until the distant future) Nov. 6, 2013S&OC Data Management System Design Review 3

40. 5-40 Imaging Data Products Generate most combined/least combined data products Least combined – exposures processed through level 2b Most combined: Mosaic, Dither, etc. Generate level 3 data products directly from all relevant level 2b exposures Not from any intermediate data products Separate observation products generated for Each SI configuration (e.g., filter) NIRCam LW and SW channels Nov. 6, 2013S&OC Data Management System Design Review 3

41. 5-41 Dither Data Products Dither data product generated if no mosaic May extend over multiple visits. No intermediate data products are generated from sub-pixel dithers. All exposures of the same target using the same filter combined into a single dither data product. All four NIRCam SW detector exposures combined into dither data product. LW detector exposures combined into separate data product. Nov. 6, 2013S&OC Data Management System Design Review 3

42. 5-42 NIRCam Multi-Visit Dither Nov. 6, 2013S&OC Data Management System Design Review 3

43. 5-43 MIRI Imaging Dither Patterns The distribution of sampling positions for the Cycling, 12- point Reuleaux, and 5-point Gaussian patterns, shown at medium scale. The origin is the aperture center, where the target is specified. Sub-pixel sampling may also be included. Dither pattern size depends on filter. Nov. 6, 2013S&OC Data Management System Design Review 3

44. 5-44 Mosaic Data Products Mosaic data products combine all possible pointings in an observation. A mosaic is typically completely defined within an observation. Except mosaic can be split into two observations if Not all tiles have guide stars Multiple position angles improve observing efficiency The first observation fails No intermediate data products generated (e.g., dither data products). Do not extend mosaics over programs in the pipeline. Standard pipeline does not address higher level data products. Nov. 6, 2013S&OC Data Management System Design Review 3

45. 5-45 Mosaic Observation Courtesy of Jeff Valenti Nov. 6, 2013S&OC Data Management System Design Review 3

46. 5-46 Higher Level Data Products Pipeline Association Baseline: Combine mosaics/dithers for a single observation (Level 3 data product) Higher (> Level 3) data products build by associating data outside routine science data processing pipeline Mechanisms and tools to identify exposures to associate and facilitate combining exposures will help Example: Multiple observations from a single target in a program Nov. 6, 2013S&OC Data Management System Design Review 3

47. 5-47 Future Data Product Design Work Nov. 6, 2013 S&OC Data Management System Design Review 3

48. 5-48 Future Data Product Design Work Continue to develop association mechanisms and level 3 imaging data products (DMS Build 3) Association design review in 2014 Add keywords derived from observatory ephemeris to level 1b headers (DMS Build 3) Compute differential velocity aberration correction Develop FGS guide star image/jitter pipeline and data products (DMS Build 4) Add keywords from DMS engineering database to level 1b headers (DMS Build 4) Develop level 2 spectral data products (DMS Build 5) Data array scheme for spectral data Spectral association mechanisms and level 3 data products (DMS Build 6) Nov. 6, 2013S&OC Data Management System Design Review 3