Download presentation
Presentation is loading. Please wait.
1
Enhanced File Consistency Checking
ADMT #10 – Toulouse, France 30 September - 2 October 2009 Mark Ignaszewski FNMOC ADMT September – 3 October 2009
2
ADMT-10 30 September – 3 October 2009
Background 2 types of format and consistency checks failures: Errors: These block distribution of the data on the GDAC until corrected. Warnings: Things we’d like to see corrected but data is distributed on the GDAC as is Unless otherwise noted, the tests generate errors and block data distribution ADMT September – 3 October 2009
3
ADMT-10 30 September – 3 October 2009
Basic Format Checking Basic format checking has not changed All file types are checked to ensure the Dimensions, Variables, and Attributes conform to the Argo specification Including “highly-desirable parameter” checks All strings checked for NULL characters (Warning) ADMT September – 3 October 2009
4
Enhanced Profile Checking
Meta-data checks Date Checks QC code checks <PARAM> variable checks D-mode specific file checks ADMT September – 3 October 2009
5
ADMT-10 30 September – 3 October 2009
Meta-data Checks PLATFORM_NUMBER: 5 or 7 numeric digits (second digit “9” for 7) DATA_STATE_INDICATOR: One of the recommended codes from reference table 6 DIRECTION: ‘A’ or ‘D’ DATA_CENTRE: Valid for the DAC DATA_MODE: ‘A’, ‘D’, or ‘R’ INST_REFERENCE: Set (Warning) POSITIONING_SYSTEM: Set (Warning) WMO_INST_TYPE: Valid (ref table 8) (Warning) ADMT September – 3 October 2009
6
ADMT-10 30 September – 3 October 2009
Date Checks All dates are checked for validity and consistency String date settings checked for validity DATE_UPDATE / DATE_CREATION / HISTORY_DATE / CALIBRATION_DATE 14 digit strings; valid (e.g., seconds must be 0 to 59) DATE_UPDATE and DATE_CREATION must be set ADMT September – 3 October 2009
7
ADMT-10 30 September – 3 October 2009
Consistency of Dates Jan 1, 1995 JULD Within 12 hours JULD_LOCATION DATE_CREATION HISTORY_DATE No order imposed CALIBRATION_DATE DATE_UPDATE Within 2 days (Warning) GDAC file time ADMT September – 3 October 2009
8
ADMT-10 30 September – 3 October 2009
QC Code Checks JULD_QC and POSITION_QC: Valid values ADMT September – 3 October 2009
9
ADMT-10 30 September – 3 October 2009
<PARAM> Checks STATION_PARAMETER: Only valid parameter names No “blank” entries No duplicate entries PRES, TEMP, PSAL are required Check that the <PARAM> variables exist for every STATION_PARAMETER Check that no other <PARAM> variables (with data) exist in the file. If mode = ‘A’ or ‘D’: Check that all <PARAM>_ADJUSTED have data Subject to the D-mode “QC=4” rules in the QC manual ADMT September – 3 October 2009
10
<PARAM> Checks (continued)
<PARAM>_QC and _ADJUSTED_QC Only valid QC codes - No “fill values” Missing data flagged with 0, 4, 9 Real-time profiles: Only codes 0 through 4 Required parameters (PRES, TEMP, PSAL): Cannot be code 0 if data is not missing PROFILE_<PARAM>_QC: Valid value Correct value Check that N_PARAM and N_LEVEL are not larger than necessary (Warning) ADMT September – 3 October 2009
11
ADMT-10 30 September – 3 October 2009
D-mode File Checks DATA_MODE = “D” DATA_STATE_INDICATOR = “2C” or “2C+” Same <PARAM> in PARAMETERS as in STATION_PARAMETERS If PRES_ADJUSTED_QC = 4, TEMP_ADJUSTED_QC and PSAL_ADJUSTED_QC = 4 *_ADJUSTED = missing ADMT September – 3 October 2009
12
ADMT-10 30 September – 3 October 2009
D-mode File Checks SCIENTIFIC_CALIB_COMMENT and CALIBRATION_DATE set for every <PARAM> and N_CALIB At least one HISTORY record HISTORY_INSTITUTION and _DATE set Plus, the <PARAM> and date checks previously discussed ADMT September – 3 October 2009
13
ADMT-10 30 September – 3 October 2009
Results Tested every cycle with a 1 or 5 in the ten’s digit 015, 055, 115, 155 Warnings 9 DACs have problems with NULLs in strings KORDI seems to be OK Some only in few variables, some in many variables A couple “N_LEVELS too large” KORDI sets the variable larger than necessary a lot “INST_REFERENCE” not set 1 JMA file ADMT September – 3 October 2009
14
ADMT-10 30 September – 3 October 2009
Results: Date checks JULD after DATE_CREATION Coriolis – only a few files INCOIS – many files – large time differences HISTORY_DATE and/or CALIB_DATE after DATE_UPDATE CSIO – Many files – Big time differences Invalid dates: AOML, INCOIS: Bad values MEDS: Too short (missing seconds) ADMT September – 3 October 2009
15
Results: <PARAM> checks
‘A’ or ‘D’: *_ADJUSTED not set Identified some missing *_ADJUSTED data Was not handling the “QC = 4” rule correctly PROFILE_<PARAM>_QC: Incorrect values AOML, Coriolis, CSIO, JMA, MEDS Missing variables CSIRO, INCOIS (DOXY_ADJUSTED_ERROR) <PARAM>_QC and *_ADJUSTED_QC Numerous inconsistencies reported Some illegal values ADMT September – 3 October 2009
16
Results: D-file checks
DATA_STATE_INDICATOR: Coriolis and MEDS: Question about “2C+” A few MEDS files set to “2B” ADMT September – 3 October 2009
17
Results: D-file checks
PARAMETER and SCIENTIFIC_CALIBRATION_* PARAMETER or CALIB_DATE not set for many files PRES, TEMP: AOML, CSIO, MEDS TEMP, CNDC: CSIRO PRES, TEMP, PSAL: JMA SCI_CALIB_COMMENT not set for many files PRES, TEMP: JMA N_CALIB larger than necessary in many files BODC (many files), Coriolis (few files) No calibration information in some D-files: Coriolis ADMT September – 3 October 2009
18
ADMT-10 30 September – 3 October 2009
Plan Implement in routine processing AS ADVISORY 20 Oct 2009 Transition to IFREMER: Oct-Nov 2009 Implement as operational checker: end-Nov 2009 ADMT September – 3 October 2009
19
ADMT-10 30 September – 3 October 2009
Still Needed <PARAM>_ADJUSTED_ERROR: Set Cross-file checks: Cycle-to-cycle: Consistent positions and times Duplicates Meta-data file comparisons Greylist: Should QC be checked? ADMT September – 3 October 2009
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.