1. Methodology for Art. 12 assessment of 2018 Art 8-9-10 updates
Melanie Muro
WK DIKE, 1 October 2018, Brussels

2. Outline
Set out the overall approach proposed for the assessment of the 2018 reports:
Key assessment questions;
Sources of information;
Steps, tools and methods.
Outline the Member States’ role in ensuring a comprehensive and accurate technical assessment of their Articles 8, 9 and 10 reports.
The assessment approach presented today is work in progress.
Art 8.1c assessment – under construction
Highlight
The assessment approach presented today in is still being developed;
Some of the assessment questions, assessment criteria and methods might therefore change as the methodology is further refined and tested over the coming months.

3. Assessment scope and objectives
The technical assessment will cover the Articles 8, 9 and 10 updates that are due to be reported by the Member States by 15 October 2018.
The purpose of the assessment is two fold:
To assess the adequacy, consistency and coherence of the 2018 updates, and
To identify progress in the implementation of the MSFD by comparing between and 2012 assessment results.

4. Broad assessment criteria
Technical assessment will cover the same broad criteria as the previous Article 12 reporting round, focusing on:
Adequacy, meaning the extent to which the reported information is good enough to meet the objectives of the Directive and the technical requirements of Articles 8, 9 and 10;
Consistency, meaning the extent to which the information reported for Articles 8, 9 and 10 is internally consistent among the three articles (assessment, determination of GES and the setting of environmental targets);
Regional coherence, meaning the extent to which information reported by one Member State aligns with the information submitted by the other Member States in the same marine region or sub-region.
Adequacy
Assessing adequacy means looking at the content of the reported information; hence it needs more specific assessment questions. It involves evaluating whether the information notified is good enough to meet the objectives of the Directive and the technical requirements of Articles 8, 9 and 10.
For example, the adequacy assessment will consider whether the definition GES is specific and clear and if, overall, GES as determined by the Member State is sufficient to achieve or maintain good environmental status of marine waters across the EU by GES is to be determined through a set of criteria and methodological standards for all descriptors’ criteria elements as laid down in Commission Decision (EU) 2017/848 of 17 May 2017 on criteria and methodological standards on good environmental status of marine waters. It is necessary to consider whether all descriptors are covered or not and whether the omission of specific criteria is clearly explained and reasonable or not. The assessment will also check whether, when threshold values are used for GES, they are clear and specific.
Consistency
Consistency refers to the relationship between Articles 8, 9 and 10 within an individual Member State’s Marine Strategy. It implies that the different elements notified by one Member State connect clearly together, do not conflict each other and that there is no gap in addressing the pressures and impacts identified. For example, has an environmental target been established under Article 10 for a problem that has been identified in the assessment under Article 8 (i.e. where GES has not yet been achieved)?
Coherence
The assessment of the status of the marine waters, the determination of GES, and the setting of environmental targets and associated indicators must be carried out in the framework of the requirement for regional cooperation, keeping in mind the objective of coherence of frameworks within the different regions (Art. 5(2) and Art. 6) and coherence across the EU (Art. 12).
Therefore, coherence refers to the relationship between the reports of different Member States, firstly, within one marine region or sub-region and, secondly, across the Union. It implies that the elements notified, status assessment, GES definitions and targets (e.g. the coverage, the scale and the level of detail) should be similar (in the case of GES determinations) or comparable (in the case of targets and status assessments) and do not contradict each other at these different geographical scales. The Article 12 assessment  allows for comparison between marine regions or sub-regions and therefore is an opportunity to strengthen coherence at EU level in the implementation of the Marine Strategy Framework Directive.
Mention that there will also be an EU-level/inter-regional comparative analysis/overview.

5. Assessment questions and criteria
Broad assessment criteria
Adequacy
Coherence
Consistency
Assessment questions
Per article (and descriptor)
Assessment criteria
Operationalise questions and guide technical assessment
Explain
The consultants, in close collaboration with the Commission, have defined a list of assessment questions for each of the three broad criteria to structure the evaluation. These questions broadly address how the technical and legal provisions of the Directive have been implemented. For each of the assessment questions, a set of assessment criteria has been identified to guide the technical assessment and to ensure that a consistent approach is followed across all Member State assessments.
Table 1 in the methodology paper provides an overview of the assessment questions and assessment criteria per article and broad criterion. For each assessment question, the table details the data needed to carry out the assessment by identifying the XML schema fields which will provide the most critical information necessary for the Article 12 assessment. At present, the table provides a generic list of assessment questions and criteria which can be tailored to the different descriptors if necessary.

6. An example Broad assessment criteria Assessment questions
Relevant data
(Schema field )
Article 9: GES determination
Adequacy
To what extent are the 2017 Decision criteria covered?
Number of primary criteria used by the Member States
Number of secondary criteria used
Clear and grounded justification provided in case of non-use
GEScomponent
Justification NonUse
Coherence
To what extent do Member States in one marine (sub-) region use the same primary/secondary criteria?
All Member States in the region or subregion use the same primary and secondary criteria
Primary criteria used
Secondary criteria used as alternative to primary
Mention: Table 1 in methodology paper
Broad assessment criteria
Assessment questions
Assessment criteria

7. Scoring and aggregation
Each assessment question is scored using a 4-point scale (e.g. ranging from adequate, to mostly adequate, mostly inadequate and inadequate).
A detailed justification for the score will be provided by the expert carrying out the assessment.
Scores will be weighted and aggregated across the assessment questions up to descriptor level to arrive at overall assessment of adequacy, consistency and coherence of the information reported by the Member States.
Explain
A scoring system will be developed where each assessment question is scored using a 4-point scale (e.g. ranging from adequate, to mostly adequate, mostly inadequate and inadequate) based on the degree to which the assessment criteria are met by the reported information. A detailed justification for the score will be provided by the expert carrying out the assessment. To arrive at an overall assessment of adequacy, consistency and coherence of the information reported by the Member States, the scores will be weighted and aggregated across the assessment questions up to descriptor level. 

8. A hypothetical example
Assessment questions
Conclusions
Weighting
Score per questions
To what extent are the 2017 Decision criteria covered?
Adequate = 4 40
To what extent is the quantitative determination of GES adequate?
Mostly adequate = 3 20 15
To what extent are levels of ambition defined for GES adequate?
Mostly inadequate = 2 10
To what extent do Member States in one marine (sub-) region use the same primary/secondary criteria?
Inadequate = 1 5
Overall score in %
(out of 100)
100 70
Overall conclusions 2018
Mostly appropriate

9. Assessment of progress
Technical assessment will NOT look at information notified to the Commission in 2012
Comparison of results of 2012 and 2018 technical assessment at descriptor level.
Assessment questions
Conclusions
Weighting
Score per question
To what extent are the 2017 Decision criteria covered?
Adequate = 4 40
To what extent is the quantitative determination of GES adequate?
Mostly adequate = 3 20 15
To what extent are levels of ambition defined for GES adequate?
Mostly inadequate = 2 10
To what extent do Member States in one marine (sub-) region use the same primary/secondary criteria?
Inadequate = 1 5
Overall score in % (out of 100)
100 70
Overall conclusion 2018
Mostly appropriate
Overall conclusion 2012
Mostly inadequate
Change since 2012
+ 1
Explain
In order to identify the level of progress since Member States first reported on the implementation of Articles 8, 9 and 10, the scoring results of the 2012 and 2018 technical assessments per descriptor will be compared. The assessment will not look at the information notified to the Commission in 2012, only at the assessment results.
 We agreed at meeting on 13/9 to not have this intermediate aggregation

10. Data to be used in the assessment
Technical assessment will primarily draw from information reported via XML reporting sheets:
Data to be used for each assessment question detailed in data request sent to Member States on 12 July as well as methodology paper;
Electronic reporting and use of standardised format ensures higher accuracy and consistency of technical assessment;
Technical assessment can be conducted more efficiently.
Emphasise
Member States are expected to submit their Art 8-10 updates via the XML reporting sheets into the EEA's ReportNet facility. Some Member States have however indicated that they will also submit one or more text-based reports ('paper' reports) covering the three articles, collectively or separately. The latter are submitted as PDF files, also via the EEA's ReportNet facility.
The reporting sheets have been developed in consultation with Member States (WG DIKE) and provide the specifications for the content of the reports, as described in the reporting guidance. They are converted into a database tool and then schemas, which enable the information to be captured in standardised formats (e.g. use of specified data formats and term lists) for upload into the ReportNet system, which facilitates an electronic and automatic analysis of the information. These reporting sheets aim to capture the information in a structured and standardised manner across all Member States, which greatly facilitates their analysis and comparison.
Given the limited timeframe for completing and publishing the Article 12 assessment of the submitted information and for the Commission to prepare its report (6 months following submission of all reports, according to Article 12), it is particularly important for the Commission that information is submitted electronically in a precise and consistent manner across the Member States, as described in the reporting guidance.

11. Assessment tools
The assessment will be conducted using an online Assessment Module. The Module will (eventually):
Present Member State reporting information for Articles 7 -11, 13 and 14, 18 and 19(3) and subsequent 6-yearly Article 17 updates for Articles 8, 9, 10, 11, 13 and 14.
Integrate Member State information with Commission assessments for MSFD Article 12 (for Art. 8, 9, 10 and 11) and 16 (for Art. 13 and 14).
Organise and present information per Member State, per (sub)region and at EU level, via differing levels of detail.
Functionalities for Phase 1:
Present Member State information as reported for Art in 2018 (raw xml information, as well as previous 6-year cycle (i.e reports); hyperlink to text-based reports.
Provide machine translation to English for text fields, where needed.
Provide aggregated reports of Member State information via a dashboard as charts, tables, maps; (e.g. number of targets or measures per descriptor).
Provide assessment reports via specified fields.
Provide report export functions e.g. to pdf and Word, to enable publication of the assessments .
Explain
The assessment will be conducted using an online Assessment Module which will harvest the required information from the Member State reports submitted to ReportNet and hold all the necessary Article 12 assessment information for the 2018 reports.
To present Member State reporting information for Articles 7 (competent authorities), 8 (initial assessment), 9 (determination of GES), 10 (environmental targets), 11 (monitoring programmes), 13 (programmes of measures), 14 (exceptions), 18 (interim reporting on PoMs) and 19(3) (access to data) and subsequent 6-yearly Article 17 updates for Articles 8, 9, 10, 11, 13 and 14.
To integrate Member State information with Commission assessments for MSFD Article 12 (for Art. 8, 9, 10 and 11) and 16 (for Art. 13 and 14).
To organise and present information per Member State, per (sub)region and at EU level, via differing levels of detail.
To allow a comparison between the results of the 2012 and 2018 assessments, both the data reported in 2012 as well as the assessment results (published in 2014) will be integrated into the module.

12. Process
The assessment will be done by technical experts, by answering a set of assessment questions and will be carried out in several steps:
Assessment of adequacy and consistency of the reported information.
Assessment of coherence once all Member States within one marine (sub)region have reported.
EU-level comparative analysis.
NO general factual correctness check by Member States is planned, as
All necessary checks are done at XML upload stage.
Standardised format/type of information reported reduces the amount of interpretation required by technical assessors.
Emphasise
It is expected that the Member State reports will contain a large amount of quantitative data and follow the structure and format provided by the reporting tools (and guidance). This should allow a much more accurate technical assessment of the reported information and should therefore reduce any ambiguities and thus the need to interpret the information. Against this background, the technical assessment will proceed without a general factual correctness check by the Member States in favor of a more targeted consultation with individual Member States should the need for clarification arise.

13. Outputs Different types of output will be produced:
This assessment exercise (on 2018 reports) and its outputs present an important first step in developing and populating a comprehensive database allowing the Commission to more effectively monitor and assess progress in MSFD implementation;
Technical reports (to be generated automatically by exporting the assessment results as a pdf file);
In put to the Commission’s guidance to Member States as per Article 12;
Input to the Commission’s evaluation of the implementation of the Marine Directive (Article 20 (3)(b)) which is due in 2019.
Explain
These technical reports will be shared with the Member States and published, as has been the practice with previous Article 12 and 16 assessments. These technical assessments will be used by the Commission to provide any guidance considered necessary to Member States, in accordance with its obligations under Article 12.

14. Timeline and Progress Timeline MS reporting due mid October 2018;
Assessment reports to be produced within 6 months of reviewing all Member States reports.
Progress and next steps
Assessment questions and criteria
Prefilling of Article 12 assessment data from 2012
Contribute to development of Assessment Module
Finalise draft of guidance for assessors (September – October)
Pilot assessment methodology and module (October – December)

15. Questions or comments?
Melanie Muro