File Formats File Name Conventions File Contents File Sizes

Slides:



Advertisements
Similar presentations
A NEW CTE PHOTOMETRIC CORRECTION FORMULA FOR ACS Marco Chiaberge TIPS meeting 05/16/2012.
Advertisements

Geoprocessing; Useful Tools You Should Know in ArcToolbox Unlock the hidden secrets of ArcToolbox to discover tools that make your work easier and analysis.
An introduction to MEDIN Data Guidelines. What MEDIN data guidelines are not… Protocols for collection methods Prescriptive of how you have to collect.
Discussion of CalWebb contents M. Robberto (facilitator)
Proposed CalJWST Structure M. Robberto. Cross-Instrument Rationale 2 level processing  CALinsA: from RAMP to calibrated images  CALinsB: associated.
HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.
Chapter 3: Introducing the Microsoft.NET Framework and Visual Basic.NET Visual Basic.NET Programming: From Problem Analysis to Program Design.
Understanding of Automation Framework A Storehouse of Vast Knowledge on Software Testing and Quality Assurance.
Hubble Legacy Archive Lee Quick - TIPS meeting July 19, 2012 Goals Data History Current Work Demo.
Biostatistics Analysis Center Center for Clinical Epidemiology and Biostatistics University of Pennsylvania School of Medicine Minimum Documentation Requirements.
Pipeline calibrations of ACS data Max Mutchler Hubble Space Telescope Calibration Workshop October 2005.
Sampling January 9, Cardinal Rule of Sampling Never sample on the dependent variable! –Example: if you are interested in studying factors that lead.
6e-1 Science Data Products Daryl Swade DMS Systems Engineer S&OC System Design Review #1.
PACS IBDR 27/28 Feb 2002 SPU High Level Software1 H. Bischof, A.N. Belbachir (TUVIE) F. Kerschbaum, R. Ottensamer, P. Reegen, C. Reimers (UVIE)
NGC 6217 in DSI mode with F658N Mosaic before new superbias why we don’t release images until after SMOV! ACS CCD monitoring and pipeline calibration review.
File FormatsFile Formats File Name ConventionsFile Name Conventions File ContentsFile Contents Association Contents & RulesAssociation Contents & Rules.
Data Management Subsystem Jeff Valenti (STScI). DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS.
An introduction to MEDIN Data Guidelines. What MEDIN data guidelines are not… Protocols for collection methods Prescriptive of how you have to collect.
1 The WFC3 Quicklook Project 17 January What is Quicklook? The WFC3 Quicklook project is a complete archive of all ~90k in-flight WFC3 images.
1 STScI Pipeline Thermal-vacuum data handling I. Neill Reid for WFC3 Instrument CALibration team ICAL.
The WinMine Toolkit Max Chickering. Build Statistical Models From Data Dependency Networks Bayesian Networks Local Distributions –Trees Multinomial /
Keyes - 5 Mar 2001 Slide 1 of 16 COS Science Team Meeting – STScI Status  STScI Integrated Scheduling System Development – TIME-TAG (primary science mode)
+ AstroDrizzle Products for HST & JWST 2014 STScI Calibration Workshop Jennifer Mack, STScI.
NPSG Data Exchange Where We Are Today and What We Need David M. Rapoport, MD NYU School of Medicine Philadelphia January 27, 2006.
COS PIPELINE PDR Daryl Swade December 7, 2000OPUS / OTFR Space Telescope Science Institute 1 of 24 Science Data Processing
ARCSS Data Management Support Overview and Update James Moore Steve Williams NCAR Earth Observing Laboratory 3-5 October 2007.
ACS Drizzling Overview J. Mack; DA Training 10/5/07 Distortion Dither Strategies MultiDrizzle ‘Fine-tuning’ Data Quality Photometry.
WFPC2 UPDATE TIPS : August 15, 2002 L.M. Lubin New WFPC2 Documentation 1.Cycle 12 Instrument Handbook (V7.0, Biretta et al.)  Updated information on the.
Chapter 13 – C++ String Class. String objects u Do not need to specify size of string object –C++ keeps track of size of text –C++ expands memory region.
SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline Calibration Goals of CALCOS Association Table Input and Output Files High Level.
1 A Balanced Introduction to Computer Science David Reed, Creighton University ©2005 Pearson Prentice Hall ISBN X Chapter 4 JavaScript and.
UPDATED CTE CORRECTION FORMULAE FOR ACS Marco Chiaberge Pey Lian Lim, Vera Kozhurina-Platais, Marco Sirianni Ron Gilliland, Jennifer Mack.
28 February 2008 COS Training Series IV 1 COS Training Series IV: COS Post-observation M.E. Kaiser P. E. Hodge 28 February 2008 COS.
SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline PDR File Structure Overview Charles D. (Tony) Keyes 7 December 2000.
NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives.
Overall ArchitectureOverall Architecture Use of AssociationsUse of Associations Outline of ProcessingOutline of Processing Reuse of CALACS and CALNIC CodeReuse.
Design of DSP testing environment Performed By: Safovich Yevgeny Instructors: Eli Shoshan Yevgeni Rifkin הטכניון - מכון טכנולוגי לישראל הפקולטה.
COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 12 Science Data Processing
Overall ArchitectureOverall Architecture Outline of ProcessingOutline of Processing Reuse of CALACS and CALNIC CodeReuse of CALACS and CALNIC Code Processing.
1 WFC3 Science Data Volume  Data Volume Estimates for WFC3 Operations ISR WFC ; Lisse, Henry, Knezek, Hanley; 3/27/2001 ISR WFC ; Lisse,
WFC3 PIPELINE CDR Jim Rose October 16, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 13 Science Data Processing
COS OPUS/HDA CDR Daryl Swade July 23, 2001Keywords and Associations Space Telescope Science Institute 1 of 14 Keywords and Associations.
CUPID (Bob Narron)GRITS - May 14, CUPID “Customizable User Pipeline for IRS Data” Abstract: The CUPID package will allow users to run the Spitzer.
STScI TIPS, 15 January 2004MultiDrizzle Overview - Anton Koekemoer1 MultiDrizzle Status and Development Overview Anton Koekemoer, ACS+WFPC2 Branch Project.
WAVELET TUTORIAL GENERATION OF DESIGNATURE OPERATORS.
Data Pipeline Wei Zheng Johns Hopkins University.
The CORNISH data reduction pipeline. Architecture - Overview AIPS Tasks Obit Tasks ObitTalk / Python Control Scripts MySQL Database Book-keeping Raw Data.
HST and JWST Pipelines and Reference Files
DADS Ingest and Distribution Support for WFC3 Daryl Swade
An introduction to MEDIN Data Guidelines September 2016
Chapter 7 Text Input/Output Objectives
Chapter 7 Text Input/Output Objectives
Understanding of Automation Framework
Chapter 7 Text Input/Output Objectives
New static DQ masks for NICMOS
The aXe slitless extraction software
An introduction to MEDIN Data Guidelines.
FITS Header Keywords for ground test data
JavaScript: Functions.
HDF5 format at European XFEL
The art of imaging with Hubble
PDAP Query Language International Planetary Data Alliance
Functional Units.
Detector Parameters Marco Sirianni - ESA.
SETL: Efficient Spark ETL on Hadoop
WFC3 Pipeline Critical Design Review October 16, 2001
Chap. 7 Regularization for Deep Learning (7.8~7.12 )
Observing with Modern Observatories (the data flow)
JavaScript: Arrays.
EXP file structure.
Presentation transcript:

File Formats File Name Conventions File Contents File Sizes WFC3 Dataset Format File Formats File Name Conventions File Contents File Sizes

File Format Overview File formats utilize STIS, NICMOS, and ACS heritage Run-time and archive formats are FITS with image and table extensions Images represented by several data arrays, collectively known as an IMSET (image set) 2 UVIS chip images stored as 2 IMSETs in one file IR MultiAccum readouts stored as multiple IMSETs in one file Individual exposures – including members of CR-Splits, Repeat-Obs, and Dithers – stored in separate files CR-Splits, Repeat-Obs, and Dithers are associated Mosaic members (specified across >1 exposure logsheet line) are not associated February 13, 2001 WFC3 Data Formats

WFC3 will retain ACS and some NICMOS name conventions WFC3 File Names WFC3 will retain ACS and some NICMOS name conventions Suffix Type Contents Cal Class _raw Image Raw science image Uncal _asn Table Association table Uncal,Cal _spt Support file _trl Trailer file _ima Intermediate IR MultiAccum image Cal _flt Calibrated image (UVIS CRSplit=0) _crj CR-cleaned calibrated image (UVIS CRSplit>0 or IR) _sfl Repeat-Obs summed image _dth Dither combined image February 13, 2001 WFC3 Data Formats

Science File Contents IMSET contents UVIS (follows STIS and ACS) Science Image (SCI) Error Image (ERR) Data Quality Flags (DQ) IR (follows NICMOS) Number of Samples (SAMP) Exposure Time (TIME) Primary Header SCI Extension ERR Extension DQ Extension x 2 IMSETs Primary Header SCI Extension ERR Extension DQ Extension SAMP Extension TIME Extension x NSAMP IMSETs February 13, 2001 WFC3 Data Formats

Science File Contents (cont.) RAW UVIS: 2 raw IMSETs from one exposure (1 IMSET per chip) IR: NSAMP raw IMSETs from one exposure IMA IR only: NSAMP calibrated IMSETs from one exposure FLT UVIS only: 2 calibrated IMSETs from one (non-CRSPLIT) exposure CRJ UVIS: 2 calibrated and CR-cleaned IMSETs from multiple CR-SPLIT exposures IR: 1 calibrated and CR-cleaned IMSET from one exposure (CR rejection accomplished via MultiAccum readout combination) February 13, 2001 WFC3 Data Formats

Science File Contents (cont.) SFL UVIS: 2 calibrated and summed IMSETs from Repeat-Obs exposures IR: 1 calibrated and summed IMSET from Repeat-Obs exposures DTH UVIS and IR: 1 calibrated and drizzled IMSET from Dithered exposures February 13, 2001 WFC3 Data Formats

Follows ACS Conventions Association Tables Follows ACS Conventions MEMTYPE* Description EXP-CRJ Input exposure for single CR-combined product PROD-CRJ Single CR-combined output product EXP-CRn Input exposure for CR-combined product n PROD-CRn CR-combined product n EXP-RPT Input exposure for single Repeat-Obs combined product PROD-RPT Repeat-Obs combined output product EXP-RPn Input Repeat-Obs exposure n PROD-RPn Repeat-Obs combined product n EXP-DTH Input exposure for Dither-combined product PROD-DTH Dither-combined output product *CR-Splits and Repeat-Obs are mutually exclusive within a single association February 13, 2001 WFC3 Data Formats

Sample Association Tables CR-SPLIT=3, No Dither CR-SPLIT=2, 2-position Dither MEMNAME MEMTYPE J1xx52ecm EXP-CRJ J1xx52egm J1xx52emm J1xx52011 PROD-CRJ MEMNAME MEMTYPE J1xx52ecm EXP-CR1 J1xx52egm J1xx52011 PROD-CR1 J1xx52emm EXP-CR2 J1xx52eom J1xx52012 PROD-CR2 J1xx52010 PROD-DTH February 13, 2001 WFC3 Data Formats

Sample Association Tables 4-position Dither, CRSPLIT=REPEATOBS=0 MEMNAME MEMTYPE J1xx52ecm EXP-DTH J1xx52edm J1xx52eem J1xx52efm J1xx52010 PROD-DTH February 13, 2001 WFC3 Data Formats

Trailer File Follows ACS Design CALWF3 modules write to explicitly-chosen trailer files Pre-CALWF3 modules (e.g. Generic Conversion, WCS) create and populate individual trailer file for each exposure CALWF3 modules that process individual exposures append to existing individual trailer files CALWF3 modules that process multiple exposures append to concatenated product trailer file (contains log for all members and products) February 13, 2001 WFC3 Data Formats

Keywords UVIS and IR images will use separate sets of keywords Reuse ACS WFC for WFC3 UVIS and NICMOS for WFC3 IR keywords, as appropriate Minor differences between ACS WFC and NICMOS keyword conventions (e.g. “dqicorr” vs. “maskcorr”) resolved by adopting ACS WFC conventions for both UVIS and IR images Keeps WFC3 UVIS consistent with STIS and ACS Keeps WFC3 UVIS consistent with WFC3 IR Results in minor differences between NICMOS and WFC3 IR February 13, 2001 WFC3 Data Formats

WFC3 File Sizes UVIS IR File Type Size (MB) ASN Table <0.1 SPT Image ~0.1 TRL RAW 34 FLT 168 CRJ SFL DTH File Type Size (MB) ASN Table <0.1 SPT Image <1 TRL RAW 33 (max) 1 IMA 168 (max) 2 CRJ 17 SFL DTH 1 2 MB per readout 2 10.5 MB per readout February 13, 2001 WFC3 Data Formats