1. Doc MET 400 (2015) & Data validation Training course
Training course on European Union Trade in Goods Statistics

2. Training course on European Union Trade in Goods Statistics
Table of content
Introduction
Doc MET400 : what is it ?
Link with legislation
Versions
Structure of Doc MET 400 document
Contact persons
Transmission of detailed trade data
General structure
Logical format
Physical format
Non confidential data
Confidential data
Reception of detailed trade data
Validation
Training course on European Union Trade in Goods Statistics

3. Training course on European Union Trade in Goods Statistics
Table of content
Introduction
Doc MET400 : what is it ?
Link with legislation
Versions
Structure of Doc MET 400 document
Contact persons
Transmission of detailed trade data
General structure
Logical format
Physical format
Non confidential data
Confidential data
Reception of detailed trade data
Validation
Training course on European Union Trade in Goods Statistics

4. Doc MET 400 : What is it ? (1) First page :
“TRANSMISSION OF THE RESULTS OF INTRA AND EXTRA-COMMUNITY TRADE
The purpose of this paper is to determine the rules for the transmission of the results, which Member States are to submit to Eurostat in accordance with the regulations“
Training course on European Union Trade in Goods Statistics

5. Doc MET 400 : What is it ? (2) Rules for transmission of data
from MSs to Eurostat
Intrastat and Extrastat data
Detailed
Aggregated
“In accordance with the regulation”
Extrastat Detailed part also applies to EFTA (with some exceptions)

6. Training course on European Union Trade in Goods Statistics
Link with legislation
Legal basis :
in section 1.1 (Extrastat) and 2.1 (Intrastat) :
Extrastat Basic Regulation (EC) No 471/2009 and implementing provisions (Regulations (EU) No 92/2010 and 113/2010)
Intrastat Basic Regulation (EC) No 638/2004 and implementing provisions (Regulation (EC) No 1982/2004)
EFTA : Decision of the EEA Joint Committee No 105/2011 and 106/2011 ; Decision No 2/2010 of the European Union/Switzerland Statistical Committee
Legal basis and Doc MET 400 are complementary
Legislation ≈ methodological principles
Doc MET 400 ≈ syntactical rules <= this training
Training course on European Union Trade in Goods Statistics

7. Training course on European Union Trade in Goods Statistics
Versions (1)
Several versions ; most recent ones are :
Year 2015 : reference periods in 2015
Year 2014 : reference periods in 2014
Year 2013 : reference periods in 2013
Year 2012 : reference periods in 2012
Year 2011 : reference periods in 2011
Year 2010 : reference periods in 2010
Rev. 17 : reference periods in 2009 or 2008
Rev. 16 : reference periods in 2007
Rev. 15 : reference periods in 2006
since 2010 : yearly
Training course on European Union Trade in Goods Statistics

8. Training course on European Union Trade in Goods Statistics
Versions (2)
Version applicable :
version corresponding to reference period
=> data must be split at least on yearly basis
Revisions :
“Revisions of data referring to previous years must be sent according to the relevant revision of this document”
This training focuses on 2015 / 2014 version
Training course on European Union Trade in Goods Statistics

9. Structure of Doc MET 400 document
Part 1 : transmission format for Extrastat /Efta detailed data
Part 2 : transmission format for Intrastat detailed data
Part 3 : transmission format for aggregated data
Annexes
Training course on European Union Trade in Goods Statistics

10. Annexes 1 & 2 : physical format 6 & 7 : examples of data files
5 : summary of fields used in detailed data
3 : special product codes (chapter 99)
4 : link to a correlation table between customs procedure codes and statistical procedure (detailed data)
8 : syntax of Checklist file for detailed data
9 : syntax to request Embargo for detailed data
10 : syntax to mark detailed data as test data
11 : product codes for which Net Mass should be transmitted as zero (detailed data)
12 : nomenclature for supplementary unit codes (EFTA detailed data only)
Training course on European Union Trade in Goods Statistics

11. Training course on European Union Trade in Goods Statistics
Contact persons
For parts 1 and 2 : Mr. XAVIER RUTTEN
For part 3 : Mr. GILBERTO GAMBINI
For technical matters concerning Gesmes format :
Training course on European Union Trade in Goods Statistics

12. Training course on European Union Trade in Goods Statistics
Table of content
Introduction
Doc MET400 : what is it ?
Link with legislation
Versions
Structure of Doc MET 400 document
Contact persons
Transmission of detailed trade data
General structure
Logical format
Physical format
Non confidential data
Confidential data
Reception of detailed trade data
Validation
Training course on European Union Trade in Goods Statistics

13. General structure Logical format = fields (e.g. syntax, nomenclatures)
Physical format
= how fields
are transmitted
(e.g. positions, separators)
Parts 1 & 2 (detailed data)
Part 3 (aggregated data)
Annex 1(detailed data)
Annex 2 (aggregated data)
Training course on European Union Trade in Goods Statistics

14. Logical format - detailed data – 2015 (1)
Section Nr.
I/E/F (*)
Data type
Contents
Maximum size 1
I/E/F C
Reference period 6 2
Threshold indicator 3
Reporting country 4
Flow 5
CN product or TARIC (real code) 10
Partner country (real code) 7
Other partner country 8 E
MS where the customs declaration is lodged 9
MS of destination / MS of actual export
E/F
Statistical procedure 11
Preference 12
Mode of transport at the frontier 13
Container 14
Nationality of the means of transport
(*) I=Intrastat, E=Extrastat, F=eFta
Training course on European Union Trade in Goods Statistics

15. Logical format - detailed data – 2015 (2)
Section
Nr.
I/E/F
Data
type
Contents
Maximum size 15 E C
Internal mode of transport 1 16
Nature of transaction 2 17
Confidentiality flag 18
CN product (confidential code) 10 19
SITC product code (confidential code) 5 20
Partner countries – confidentiality flag 21
Value confidentiality flag 22
Quantity confidentiality flag 23
Supplementary units confidentiality flag 24 N
Statistical value 14 25
Quantity expressed in net mass 26
Quantity expressed in supplementary units 27 F
Supplementary unit code 28
SITC product code (real code)
(*) I=Intrastat, E=Extrastat, F=eFta
Training course on European Union Trade in Goods Statistics

16. Logical format - detailed data – 2015 (3)
All 28 fields must always be present (Intrastat / Extrastat / Efta)
Main nomenclatures :
Products :
Combined Nomenclature / Taric nomenclature
Some special codes : chapter 99, Annex 3
Countries :
Geonomenclature
Some special codes :
Nomenclatures can (and do) change !!!
Training course on European Union Trade in Goods Statistics

17. Physical format 3 formats :
fixed column width
comma delimited
Gesmes  mandatory !!!
“Exceptionally, in case of problems, and in agreement with Eurostat, Member States may temporarily use the "fixed column width" format or the "comma delimited" format”
Training course on European Union Trade in Goods Statistics

18. Gesmes (1) 1 row per data/record, containing all fields
order of fields defined by Doc MET 400
values must be left justified i.e. do not contain leading blanks or spaces
trailing blanks at the end of values must be removed
Example :
ARR :1:FR:1: :US:US:FR:FR:1:100:4:0:US:2::0:::::::5648:0:0::'
ARR :1:FR:1: :US:US:FR:FR:1:100:4:0:US:3::0:::::::47045:0:0::'
ARR :1:FR:1: :IL:IL:FR:FR:1:100:4:0:SE:2::0:::::::2632:0:0::‘ …
Training course on European Union Trade in Goods Statistics

19. Gesmes (2) Header : UNA:+.? '
UNB+UNOC:3+FR1+4D :1448+IREF GESMES'
UNH+MREF GESMES:2:1:E6'
BGM+74'
NAD+Z02+EUROSTAT'
NAD+MR+4D0'
NAD+MS+FR1'
DSI+COMEXT_EXTRA_M'
STS+3+7'
DTM+242: :203'
DTM+Z02:201501:610'
IDE+5+EUROSTAT_FT'
GIS+AR1‘
Training course on European Union Trade in Goods Statistics

20. Gesmes (3) Footer : UNT+51+MREF000001' UNZ+1+IREF000001'
Training course on European Union Trade in Goods Statistics

21. Physical format - Transmission
(sections 1.1 & 2.1)
Transmission method (sections 1.1 & 2.1) :
“via the Single Entry Point (SEP) of eDAMIS (Stadium)
dataset COMEXT_EXTRA_M (including EFTA) / COMEXT_INTRA_M
for both the first transmission of the latest month and revisions”
Checksum:
“The detailed statistics must be accompanied in the same Stadium envelop by a checklist corresponding to the total by flow of the detailed statistics for section 24 (statistical value) and section 25 (quantity expressed in net mass): see Annex 8”
Deadline :
“The EXTRA (including EFTA) / INTRA file should be transmitted no later than 40 / 70 calendar days following the reference month”
Training course on European Union Trade in Goods Statistics

22. Non confidential data Section Nr. I/E/F Data type Contents Content 15
Internal mode of transport 16
Nature of transaction 17
Confidentiality flag
Must be zero 18
CN product (confidential code)
Must be empty 19
SITC product code (confidential code) 20
Partner countries – confidentiality flag 21
Value confidentiality flag 22
Quantity confidentiality flag 23
Supplementary units confidentiality flag 24 N
Statistical value 25
Quantity expressed in net mass 26
Quantity expressed in supplementary units 27 F
Supplementary unit code 28
SITC product code (real code)
Training course on European Union Trade in Goods Statistics

23. Threshold - Extrastat Code Meaning Disseminated product code 1
data belonging to the standard category
= transmitted code 3
data for individual transactions below the statistical value threshold for which less detailed information is compiled
<Chapter>MMM000 or
99YYY000 4
adjustments for delayed or incomplete records (to be used only in exceptional cases, when customs declarations are received too late to be able to send comprehensive figures within the legal deadline)
Training course on European Union Trade in Goods Statistics

24. Threshold - Intrastat Code Meaning Disseminated product code 1
data belonging to the standard category (intra-EU traders above the exemption threshold or above the simplification threshold1, when applied), when net mass is collected
= transmitted code 2
data belonging to the simplified category (intra-EU traders below the simplification threshold and above the exemption threshold); simplified information provided for small individual transactions. 3
adjustment for trade below the exemption threshold
<Chapter>MMM000 or
99YYY000 4
adjustment for late and non-response (trade above the exemption threshold) 5
data belonging to the standard category (intra-EU traders above the exemption threshold or above the simplification threshold, when applied), when net mass is estimated 8
3+4 together [allowed only if no 3 and no 4 ; 3 or 4 to be preferred]
idem 3 or 4
Training course on European Union Trade in Goods Statistics

25. Confidential data Section Nr. I/E/F Data type Contents Content 15 E C
Internal mode of transport 16
Nature of transaction 17
Confidentiality flag
Must be 1 or 2 18
CN product (confidential code)
Must be filled in 19
SITC product code (confidential code) 20
Partner countries – confidentiality flag 21
Value confidentiality flag 22
Quantity confidentiality flag 23
Supplementary units confidentiality flag 24 N
Statistical value 25
Quantity expressed in net mass 26
Quantity expressed in supplementary units 27 F
Supplementary unit code 28
SITC product code (real code)
Training course on European Union Trade in Goods Statistics

26. Confidential data - concepts
Levels of confidentiality
Eurostat-restricted
Commission-restricted
Type of camouflage
camouflage by product
camouflage by partner
camouflage by product & partner
Indicators concerned
all indicators (value, quantity expressed in net mass, quantity expressed in supplementary units) can be declared confidential, independently ones from the others, by using a distinct flag ("confidentiality flag") for each indicator
values for indicators declared confidential will be associated to camouflaged data
other values will be associated to real (non camouflaged) data
Training course on European Union Trade in Goods Statistics

27. Levels of confidentiality : confidentiality flag
Section
Nr.
I/E/F
Data
type
Contents 17 C
Confidentiality flag
Values :
0 : confidentiality is not applied
1: access to confidential data restricted to Eurostat
2: access to confidential data restricted to Eurostat and other services within the Commission
Adjustments cannot be confidentialised
Training course on European Union Trade in Goods Statistics

28. Type of camouflage – camouflage by product
Section
Nr.
I/E/F
Data
type
Contents 5 C
CN product or TARIC (real code) 18
CN product (confidential code) 19
SITC product code (confidential code)
Section 5 : real product code (CN8 or Taric)
Section 18 : “public” product code
Same as section 5 if product is not confidential
Otherwise HS6, HS4, HS2 or
Section 19 : “public” SITC product code
to be transmitted if the information in section 18 is “poor” i.e. if HS4, HS2 or
SITC3, SITC2 or SITC1
In Extrastat : 999 also allowed if in section 18
Disseminated (=> camouflaged) data :
<HS6>SS
<HS4>S<SITC3> or <HS4>S<SITC2>S or <HS4>S<SITC1>SS
<HS2>SSS<SITC3> or <HS2>SSS<SITC2>S or <HS2>SSS<SITC1>SS
Training course on European Union Trade in Goods Statistics

29. Type of camouflage – camouflage by partner
Section
Nr.
I/E/F
Data
type
Contents 6 C
Partner country (real code) 7
Other partner country 20
Partner countries – confidentiality flag
Possible values for section 20 :
Disseminated (=> camouflaged) data :
If no confidentiality : real country codes
If confidentiality :
QY (Countries and territories not specified for commercial or military reasons in the framework of intra-Community trade)
QZ (Countries and territories not specified for commercial or military reasons in the framework of trade with third countries)
Training course on European Union Trade in Goods Statistics

30. Indicators concerned – confidentiality flags
Section Nr.
I/E/F
Data type
Contents
Example 21 C
Value confidentiality flag 22
Quantity confidentiality flag 1 23
Supplementary units confidentiality flag
Values :
0 = value of indicator is not confidential
1 = value of indicator is confidential
Example => Quantity is confidential but Value and QSU are not
Camouflage :
values for indicators declared confidential will be associated to camouflaged data
values for indicators not declared confidential will be associated to real (non camouflaged) data
=> each record is split into 2 records :
1 record with camouflaged data
1 record with real (non-camouflaged) data
Training course on European Union Trade in Goods Statistics

31. Camouflage – example - product confidentiality only (1)
real CN code=
public CN code=01
public SITC code=3
PARTNER :
real partner code=NO
public partner code=NO
real other partner code=DE
public other partner code=DE
INDICATORS
VALUE=140
NETMASS=500
QSU=111
INDICATORS CONFIDENTIALITY FLAGS
FLAG_VALUE=0
FLAG_NETMASS=1
FLAG_QSU=0
Training course on European Union Trade in Goods Statistics

32. Camouflage – example - product confidentiality only (2)
After camouflage : 2 records :
=> it is not possible to link values of confidential indicator with confidential data (real product code) :
values of the confidential indicator (net mass) is located in the first record (the one containing the camouflaged product value)
values of non confidential indicators (value, qsu) are located in the second record (the one containing confidential information)
Sum of the 2 records equal orginal record
Training course on European Union Trade in Goods Statistics

33. Camouflage – example - partner confidentiality only (1)
PRODUCT :
real CN code=
public CN code=
PARTNER :
real partner code=NO
public partner code=QZ
real other partner code=DE
public other partner code=QY
INDICATORS
VALUE=140
NETMASS=500
QSU=111
INDICATORS CONFIDENTIALITY FLAGS
FLAG_VALUE=0
FLAG_NETMASS=1
FLAG_QSU=0
Training course on European Union Trade in Goods Statistics

34. Camouflage – example - partner confidentiality only (2)
After camouflage : 2 records :
=> it is not possible to link values of confidential indicator with confidential data (real partner and other partner codes) :
values of the confidential indicator (net mass) is located in the first record (the one containing the camouflaged product value)
values of non confidential indicators (value, qsu) are located in the second record (the one containing confidential information)
Sum of the 2 records equal original record
Training course on European Union Trade in Goods Statistics

35. Camouflage – example – all confidentiality flags set (1)
PRODUCT :
real CN code=
public CN code=01
public SITC code=3
PARTNER :
real partner code=NO
public partner code=QZ
real other partner code=DE
public other partner code=QY
INDICATORS
VALUE=140
NETMASS=500
QSU=111
INDICATORS CONFIDENTIALITY FLAGS
FLAG_VALUE=1
FLAG_NETMASS=1
FLAG_QSU=1
Training course on European Union Trade in Goods Statistics

36. Camouflage – example – all confidentiality flags set (2)
After camouflage : 2 records :
=> it is not possible to link values of confidential indicator with confidential data (real product, partner and other partner codes) :
values of the confidential indicator (net mass) is located in the first record (the one containing the camouflaged product value)
values of non confidential indicators (value, qsu) are located in the second record (the one containing confidential information)
Sum of the 2 records equal original record
!!! Second record needs not to be generated !!!
Training course on European Union Trade in Goods Statistics

37. Training course on European Union Trade in Goods Statistics
Table of content
Introduction
Doc MET400 : what is it ?
Link with legislation
Versions
Structure of Doc MET 400 document
Contact persons
Transmission of detailed trade data
General structure
Logical format
Physical format
Non confidential data
Confidential data
Reception of detailed trade data
Validation
Training course on European Union Trade in Goods Statistics

38. Validation : generalities
Validation = decide whether ot not to load Data = boolean
≠ Quality measurement = assess Data quality = real number
Validation handbook
4 steps
9 rule types
4 severities
each rule is identified by a "Rule ID"
Training course on European Union Trade in Goods Statistics

39. Validation : 4 steps
Step 1 - Preliminary global data file checks: The purpose of tests of step1 is to ensure that no major inconsistency is detected at file level preventing to continue the loading of the file.
Step 2 - Records validation checks: This step is performed at record level and consists in checking the validity and consistency of the record data as well as their compliance with Doc MET400.
Step 3 - Post record check global validation: Tests in this step are performed at file level after the records have been corrected in order to evaluate if the number of corrections needed to be applied to accept the data file are within acceptable limits and therefore if the file can be loaded in the Comext databases.
Step 4 - Advanced post validation checks: These checks are performed after the data are loaded in Comext. These tests are based on more advanced statistical methods, are more time consuming and the analysis of their results cannot be automated. Example : detecting outliers in the data.
Training course on European Union Trade in Goods Statistics

40. Validation : 9 Rule types
Filling of the fields: In the Doc MET400, the field can be mandatory, mandatory depending on threshold, mandatory depending on the confidentiality flag, or optional. Errors detected in the filling of the fields will be stored here. B
Unexpected  blank
Field is provided empty whereas it should be filled in F
Unexpected Filling of the field
Filling of the field was not expected 
Formatting of the fields: This covers incorrect format of the field. It can be for example a 8 digits codes defined in the commodity field when a 10 digits code is expected. X
Incorrect format
invalid code length, alphanumeric code provided instead of numeric…
Validity: This covers the use of invalid codes not registered at all in the reference table I
Invalid Code
Code not in the reference table
Credibility C
Inter-field consistency
Inconsistency compared to DocMET400 specifications. M
Methodology
Methodological issue checks based on regulation requirement, concept of external trade. Not specifically written in the Doc MET400.
Eurostat specific rule type S
System/Treatment
No problem detected but a processing is performed to ensure consistency in the Eurostat database. N
Notes for Eurostat manager
Example : non-standard threshold given at 2 digits only) or induced error (R18 cannot be checked as R5 is in error) D
Database manager
Consistency of the Eurostat referential. Example : conversion could not be performed due to incomplete mapping of the conversion table).
Training course on European Union Trade in Goods Statistics

41. Validation : 4 Severities
Abort
Critical error (=the processing and loading cannot be further continued) E
Error
Non-critical error (=the processing and loading can be further continued) W
Warning
Warning ; further manual investigation suggested
(also covers Eurostat specific rules which are not associated to a problem within the data file but to an issue detected during Eurostat processing) I
Internal information
Information specific to Eurostat processing environment ; only associated with rules of type S, N, D
Training course on European Union Trade in Goods Statistics

42. Validation : Rule ID Rule ID = VV_XX_YZ where
VV is the variable or step concerned :
For rules in Step 1: VV=”PRE” (indicating “Preliminary global file checks”)
For rules in Step 2: VV=the field concerned (“01” to “28”)
For rules in Step 3: VV=”POST” (indicating “Post record check global validation”)
For rules in Step 4: VV=”ADV” (indicating “Advanced post validation checks”)
XX is the rule number within the section
Y is the rule type (B,F,X,I…)
Z is the severity (A, E,W, I)
Training course on European Union Trade in Goods Statistics

43. Validation : Example of rule
Rule ID = 05_02_XE
“05” indicates the rule is a rule in Step 2 concerning field number 5
“02” indicates the rule is the second rule in the list of rules in Step 2 concerning field number 5
“X” indicates the rule type is “X” (“incorrect format”)
“E” indicates the severity is “E” (“error”)
rule description : Field Commodity is only 1 digit, which is never allowed. It will be converted to 99CCC000 (or 99CCC00000 for Extrastat import) when the record is related to standard threshold, else it will be converted to 99YYY000 (00)
Training course on European Union Trade in Goods Statistics

44. Thank you for your attention !
Xavier RUTTEN
Eurostat G5
Training course on European Union Trade in Goods Statistics