1. Draft Methodology for impact analysis of ESS.VIP Projects
Pieter Everaers, Director
Eurostat Directorate A, Cooperation in the ESS, International cooperation, Resources
ESS RDG 13 June 2014

2. A methodology for impact assessment is needed
ESSnet on Standardisation :establish the business case of European standardisation initiatives
November 2012: new ESS.VIP projects have to be evaluated on the implementation costs and future benefits =>task force
May 2014: ESS Vision 2020 agreed by ESSC. Projects will be selected on criteria including cost/benefit
The ESSnet on Standardisation was launched with the aim to establish the business case of European standardisation initiatives at the strategic level
November 2012: ESSC decided that new ESS.VIP projects have to be evaluated on the implementation costs and future benefits =>creation of the TF "Impact analysis of ESS.VIP projects"
May 2014: ESS Vision 2020 agreed by ESSC. Projects will be selected on the basis of criteria. The first one is cost/benefit
ESSnet on Standardisation launched with the aim to establish the business case of European standardisation initiatives at the strategic level
ESS RDG
13 June 2014

3. Two initiatives assessed the possible approaches
Sponsorship on Standardisation (end September 2013) proposed:
Common assessment approach
Link to the life cycle of a new initiative
Structured qualitative approach
Assessment by all stakeholders in the ESS
Scoring mechanism
Basis for the European business case
Common assessment approach for all types of initiatives in the ESS
Methodology contributes to the decision steps in the life cycle of a new initiative
Structured qualitative approach with a fixed set of items
Assessment by all stakeholders in the ESS from their own perspective
Use a scoring mechanism to analyse, summarise and present the results
Use scores and comments as a basis for the European business case
ESS RDG
13 June 2014

4. Two initiatives assessed the possible approaches
Task Force "Impact analysis of ESS.VIP projects"
(April-December 2013) proposed:
Draft methodology and pilot project
Quantitative and qualitative assessments
Information collected at NSI level but impact for the ESS
Statistical and financial experts + senior management
In the quantitative phase: “benefits” = potential cost savings
SWOT based on proposal from ESSnet + flexible items
Draft methodology and pilot project (PRIX)
Combination of quantitative and qualitative assessments
Information collected at NSI level but the purpose is to assess the impact for the European Statistical System
Requires contributions and cooperation from statistical and financial experts and the involvement of NSI’s senior management
In the quantitative assessment “benefits” = potential cost savings
The qualitative assessment is a SWOT largely based on the proposal from ESSnet on standardisation but with flexible items
ESS RDG
13 June 2014

5. Reconcile the 2 methodologies
ESSC asked RDG and ITDG-DIME for a single methodology
Proposal
Combination of qualitative and quantitative analysis
2 phases of the methodology linked to the life cycle of the initiative
ESSC asked RDG and ITDG-DIME for a single methodology
Proposal
Combination of qualitative and quantitative analysis in 2 phases
Decisions around a new initiative follow a life cycle. The 2 phases of the methodology apply to specific decision points in this life cycle
ESS RDG
13 June 2014

6. Phase 1 Business case including SWOT and Vision general criteria
Phase 2 including detailed quantitative analysis

7. Phase 1 Business case includes criteria from ESS 2020 Vision
SWOT analysis based on 3 fixed items and 3 flexible items
Rating of the SWOT items
Leads to decision to launch the project or to go to phase 2
ESS RDG
13 June 2014

9. Phase 1: Vision general criteria
A. Does the proposal have a positive business case at both ESS and national levels, in the sense that benefits outweigh costs?
B. Does the proposal contribute to the needs and drivers identified in the ESS Vision such as efficiency, quality and promoting public trust?
C. Can the proposal be clearly positioned according to our ambitions for collaboration set out in the ESS Vision 2020?
D. Does the scope in that sense look manageable?
E. Is the proposal consistent with the full set of other proposals?
F. Does the proposal take into account generic European principles like proportionality and subsidiarity?
G. Is the proposal in line with the European Statistics Code of Practice?
H. Does the proposal observe existing legislation, both at national and EU level?
I. Is the proposal in line with the ESS reference architectural decision and design principles?
J. Does the proposal sustain a simultaneous production of European and national statistics?
ESS RDG
13 June 2014

10. Phase 1: List of SWOT items
1. STRENGTHS
a. Saving of resources
b. Improved process and systems quality
c. Enhanced flexibility
d. Increased harmonisation
e. Increased level of detail
f. Better availability of metadata
2. WEAKNESSES
a. Costs (of development, transition, support and maintenance)
b. Lack of agility/flexibility of the new system
c. Heterogeneity of the starting point between countries and partners
d. Complexity
e. Impact of changed process (risk of errors, discontinuity of the data)
f. Lack of appropriate skills in the NSIs
ESS RDG
13 June 2014

11. Phase 1: List of SWOT items
3. OPPORTUNITIES
a. Reduced burden on respondents
b. Enhance standardisation
c. Better communication with users and stakeholders
d. Opportunity to change to cope with resource constraints
e. Shared technical developments
f. Possibility to provide tailored feedback to data providers
4. THREATS
a. Lack of consistency with national policies (including legal)
b. Lack of support from stakeholders/financers
c. Lack of suitable skill profiles
d. Threat to data security
e. Competition from other providers
f. Dependency from external actors/providers
ESS RDG
13 June 2014

12. Phase 1: SWOT rating
ESS RDG
13 June 2014

13. Phase 2 Quantitative analysis
Focus only on business processes affected by change
Estimation of current costs, development costs and future costs of production
Cost categories: Staff , IT solutions, Overhead, Other costs
In case potential savings cannot be quantified: build on concrete experiences from countries which have modernized their production system, develop scenarios
+ Adaptation of the SWOT analysis
ESS RDG
13 June 2014

15. To be kept in mind Lessons from the pilot on quantitative approach
Analysis of current costs is possible for most countries
Data not always available for different steps of the GSBPM
Difficult for some countries to estimate their costs
=>group countries
Future costs: impact analysis of the modernization process at Member State level not systematic
An analysis of the data on current costs is possible and very fruitful for most countries
Data not always available for different steps of the GSBPM level (e.g. in countries which have already integrated their production processes)
Difficult for some countries to estimate their costs: proposal to group countries as a fall back solution
Future costs: we could not rely on impact analysis of the modernization process at Member State level since they are still not systematically done
ESS RDG
13 June 2014

16. Way forward
Proposal discussed today and in ITDG-DIME steering group 18 June
Feed the work of the high level Task Force working on the ESS Vision 2020 Implementation strategy (for DGINS)
Test on projects implementing Vision 2020 ( )
Evaluate and adapt the methodology
Draft methodology discussed today and in ITDG-DIME steering group 18 June
Output will feed the work of the high level Task Force working on the ESS Vision 2020 Implementation strategy (for DGINS Sept 2014)
Test on projects implementing ESS Vision 2020 ( )
Evaluate and adapt the methodology
ESS RDG 13 June 2014