1. I - Evidence-based Policy Making
Jean Monnet Chair
I - Evidence-based Policy Making
I.1 The role of statistics in supporting the decision making
Luigi Biggeri

2. Statistical Reasoning for decision
Steps of General Reasoning Steps of Statistical Reasoning
Problem to be solved Problem to be solved
Information needs Definition of data need
Collection of information Collection of data (Surveys)
Analysis of information Statistical Methods for Analysis
Decision (choice of the action) Decision (support choices)
Some points to be remembered

3. What to know about Statistics
Statistical Information
1 methods of data production
2 sources of statistical data
3 interpretation and comparison
Statistical Methods
1 knowledge of the methods
2 capability to implement Statistical Analysis
3 capability to interpret and present the results of the analysis

4. What to know about statistical information -a-
1 Data Production
General characteristics of the phenomenon (population) to measure
Crucial importance of the definitions (units of population and variables) to:
- conduct measures and estimations
- understand the content of the information
What is the relevant population?
What is(are) the variable(s) of interest?
Methods of producing data
- Complete or sample surveys Populations of units
- Data integration When it is not possible to have data on the population with only one survey or to obtain measure for a complex phenomenon

5. What to know about statistical information -b-
2 Knowledge of data sources and their characteristics
EU and Contry level European Statistical System (ESS) for national accounts and system of surveys (Lectures of Daniela Ghio)
Local level Usually very few data is available and may be it is necessary to use SAE to produce them (Monica Pratesi et al.)
3 Interpretation and comparisons
Interpretation Definitions used and Quality of data
Comparisons Never plain: need for standardization

6. What to know about statistical methods
1 Capability to implement Statistical Analysis
Knowledge of statistical methods
Capability to use the adequate methods for different aims
Knowledge of the intrinsic logic of the methods
2 Capability to interpret and present the results of the analysis
What the data and method used allow
Interpretation of the results obtained for the specific analysis
Methods of presentation of the statistical analyses and results (tables and graphs, ect.)

7. Few References
McClave J.T., Benson P.G. and Terry T.T., (2014), Statistics for Business and Economics, Pearson (Chapter 1, What is Statistics?)
Biggeri L, Bini M., Coli A., Grassini L and Maltagliati M., (2012), Statistica per le Decisioni Aziendali, Pearson (Chapter 1), in Italian Language

8. I - Evidence-based Policy Making
Jean Monnet Chair
I - Evidence-based Policy Making
I.2 The role of statistics to design policy intervention
a framework
Luigi Biggeri

9. Outline Introduction: importance of statistical information
Policy definition and evaluation: a statistical theoretical approach
Need for a pertinent Statistical Information System
Statistical support for the implementation of policy interventions at a local level
Different methods to get information
Need for data of high quality: how to evaluate it

10. 1 Introduction: importance of statistical information
Nobody has ever doubted that statistical information and statistical methods are indispensable for taking rational decisions
However, for a long time even policy makers have rarely and badly used the statistical information available and, at the same time, producers of official statistics have not always supplied the necessary statistical data
Now, governments at different levels are more and more interested in formalizing their decision processes and in evaluating their programmes, activities and intervention policies in economic and social areas

11. … and of a statistical information system
Obviously, this implies adequate information and, above all, specific statistical information systems and sets of indicators that official statistics should implement, for the different policies that the decision makers wish to implement
Therefore, the definition of a statistical theoretical approach for policy definition and evaluation is really important
This is important not only in the interest of the public decision-makers but also of the citizens, so that they can exercise a documented democratic control on the policy intervention

12. 2 Policy definition and evaluation: a framework
The design of political intervention is clearly a strategic field at least for three reasons
SCIENTIFIC SUPPORT FOR DECISIONAL PROCESSES
Make more adequate choices
Rationalize interventions, programs and actions
On effective implementation of the programmed targets
On impact of the implemented programs
CONTROL (Monitoring)
transparency with references to users
safeguards of society’s interests
GUARANTEE FUNCTION
To specify how policy intervention design and the ensuing impact evaluation should be organized, we can refer to the simplified framework illustrated in the following sketch

13. Design of Intervention Information on results
SIMPLIFIED FRAMEWORK OF THE DESIGN, IMPLEMENTATION AND EVALUATION OF AN INTERVENTION POLICY
FRAMEWORK
(Contest, etc.)
PROBLEM
Model/s
Design of Intervention
Information System simulation of actions and evaluation
Simulation of actions L E A R N I G L E A R N I G
Possible Consequences
Evaluation
Choice among the alternative actions
Implementation (resources and input characteristics)
Evaluation
Achievments (process characteristics)
Evaluation
Results
(Output, Outcome, Impact)
Evaluation
Information on results
Social Control

14. Framework analysis: characteristics
The sketch shows which are the steps to be implemented to define, carry out and evaluate intervention policies in a consistent framework
Framework analysis highlights some important and specific aspects that must be taken into consideration to organize policy design and evaluation. As a matter of fact, it is necessary:
To analyze the context and real situation (a good knowledge is required of how the phenomena works and how the involved units behave), and the problems that we have to face
To simulate the actions on which intervention is based with macro or micro models, to evaluate their possible consequences, and to choose among the various alternative actions

15. Framework analysis: characteristics
To evaluate each phase of implementation of the actions and the obtained results
To use results and evaluation analyses for learning aims and, if necessary, to change the plan or to improve the information system already available
To disseminate evaluation results also as a means of social control by general public and by interested bodies
Need for a specific Statistical Information System (For European Statistical System see Daniela Ghio)

16. Framework analysis: programs and effects
To develop these statistical designs it is obviously essential to deeply know the nature and characteristics of the programme
For example, it is necessary to know which are the elements that influence programme results
Obviously, the effects of the programme must be measured using response variables strictly connected to the objectives (Which poverty indicators is important to use?)
The analysis must be done considering the real operative conditions and consequently the characteristics of the decisional process (as highlighted in pict. 1).

17. Framework analysis: situation and theories
Strategic:
to define a real reference framework of the situation in which is the subject of the study and/or a working model
To know the existing economical and sociological theories
If these theories do not exist or are not convincing, it is necessary to use empirical evidence, through a sociologic, economic, managerial study of the organizations and the processes involved in the programme (in this case, the importance of the interdisciplinary cooperation is evident)

18. KEY CONDITIONS FOR THE APPLICATION OF THE SCHEME
3 Need for a Statistical Information System –a-
KEY CONDITIONS FOR THE APPLICATION OF THE SCHEME
adequate quantitative and qualitative statistical information
appropriate statistical information system for:
definition of intervention
management of intervention
evaluation of intervention
planning of quantitative and qualitative key indicators (on poverty, etc.)
analysis of characteristics of variables and indicators
INF.
METHODS:
Impact evaluation
MEASUREMENT OF NET BENEFIT OF THE INTERVENTION

19. 3 Need for a Statistical Information System –b-
To plan and evaluate the policy intervention in an objective way, it is necessary :
first “to measure”, having decided what to measure
then to have adequate measures of the variables of interest and plan the appropriate quantitative and qualitative indicators
The use of different indicators is also an opportunity to check the quality and consistency of different data on the same phenomenon as well as data on different phenomena arising from different statistical sources
In the end, the analysis of data highlights the most important results obtained from different statistical surveys (thus increasing the value added in the presentation of the results of each single survey) and provides possible solutions to problems of economic and social policy.
Numbers produced by surveys are transformed into
“political relevant information”

20. 4 Statistical support for the implementation of
4 Statistical support for the implementation of policy interventions at a local level
Figure 1 The multi-step procedure 

21. Statistical approach for selecting and analyzing Indicators
Statistical analysis helps the policy maker to identify issues, using various methods indicated in the sketch to give reply to various questions:
How is the situation of the area?
Where are the main problems?
Why did the situation and problem arisen?
in order to identify the most important aspects that need to be monitored and considered to design intervention policies
But also to reply to other questions:
Is the situation of the area improving or worsening over time?
Is there a specific trend over the last few years?
Do the sub-areas have the same characteristics and behaviour?
in order to understand where intervention is primarily required

22. The implementation of a Statistical Information System at a local level: possible general effects
Answer to the new demand of information for programming and evaluating interventions
Enhancement of the existing territorial statistical information
Increase a permanent settlement of statistical data and indicators supply on regional and sub-regional basis
Use new methodologies and tools for producing territorial statistical estimations (SAE Methods)
Disseminate results achieved to different users and scientific society as a whole

23. IMPACT OF THE DEMAND FOR TERRITORIAL AND SECTORIAL STATISTICS
Further effects of the implementation of a Statistical Information System at local level
IMPACT OF THE DEMAND FOR TERRITORIAL AND SECTORIAL STATISTICS
Increasing of sensitiveness to the topics of territorial data and indicators
Implementation of exchange of territorial data
A RELATION IMPACT
Involvement of others administrations to provide and ask for territorial data, and to disseminate the Culture of Measurement and Evaluation
AN INSTITUTIONAL IMPACT
A direct reply to an increasing demand of territorial indicators coming from other actors not directly involved in policy making
AN EXTERNAL IMPACT
Need to increase investments and financial resources to implement
the local information statistical systems

24. 5 Different methods to get information
There are various methods to collect information about large population
Advantages
Disadvantages
collection of data in a standardized methods over large number of units
Statistical surveys
Not ever it is possible to standardized; possible errors
Administrative record systems (Big Data?)
Sometimes offer very good data
Little control over the measurements
Yield deep understanding
Qualitative investigation
Small groups of informants
Observation of the behaviors of units (Big Data?)
Information on frequency of events
Are limited to a tiny fraction of behaviors (no quality control?)
Randomized experiments
Answer as stimuli cause behavior
Difficult applicability in real world
Not the only way, but the most important is Statistical Surveys
Need for Small Area Estimation Methods (SAE), Monica Pratesi Lectures

25. (Monica Pratesi Lectures)
6 Need for data of high quality: how to evaluate it
Confidence of decision makers and citizens in the quality of the statistical information is indispensable in order to use it without suspect
For this reason the international organization (UN and EUROSTAT) state that it is important that the statistical outputs meet some fixed quality standards (relevance, accuracy, timeliness, accessibility, interpretability, coherence and comparability) and actually they try to guarantee it
It is therefore necessary that the user of data know the characteristics of data and indicators and also their possible errors and interpretative limitations
The quality of data can be evaluated only from researchers that know “Methodology of Statistical Survey”
(Monica Pratesi Lectures)

26. References
Biggeri L. (2004), Official Statistics for Decision Making and Evaluation: territorial indicators, in Proceedings of OECD First Forum on: Statistics, Knowledge and Policy: Key indicators to Inform Decision Making, Palermo
Bedi T., Coudouel A. and Simler K., (2007), More than a Pretty Picture: Using Poverty Maps to Design Better Policies and Interventions, World Bank
Ray P., Greenhaigh T., Harvey G. and Walshe K., (2005), Realist Review – a new methods of systematic review designed for complex policy interventions, Journal Health Service Reseach Policy, Vol. 10, Suppl- 1 July 2005
To get more information see also the Proceedings of the OECD Forums on Statistics, Knowledge and Policy hold in Instanbul, Turkey, (2007), Busan, Korea (2009), New Delhi, India (2012) and the Forum that has been hold in Mexico from 13 to 15 October The next will be held again in Korea.