Open Days, Rome 3 July Evolution of Uval's work program: from support to evaluation Italian Evaluation Unit Open Days 3 – 5 July 2006 Laura Raimondo

Open Days, Rome 3 July Until April 2006 Ministry of Economy and Finance

Open Days, Rome 3 July Department for Development Policies CSF MA

Open Days, Rome 3 July UVAL’s tasks UVAL founded in 1982 Reformed in 1998 Re-reformed in 2002 Main tasks: –Promotion of data, information and knowledge for public policies –Support to administrations to modernize and innovate public action, –Evaluation of programs and capacity building for evaluation –Ex-ante assessement of projects

Open Days, Rome 3 July UVAL’S work program Activities (not staff) are organized in four areas –Methods and studies for public policy –Modernization and administrative innovation –Program evaluation –Project appraisal 104 activities Support to other parts of the Department and to the Network of the Italian Evaluation Units is still a large part of work (e.g., project appraisal, support to programming for )

Open Days, Rome 3 July The Evaluation Unit (UVAL) Area 1Area 2Division 3Division 4 Metodi e studi per le politiche pubbliche Trasferimento metodi e sostegno attuativo Program EvaluationProject Evaluation Tendenze e analisi delle politiche di sviluppo territoriale Sostegno attuativo ai servizi DPS e alle amm.regionali e centrali Coordination CSF evaluation system Development of guidelines and standards for the assessment of infrastructure and integrated projects Rapporto territoriale Intese e APQ, impostazione e performance. Valutazione QCS e controllo qualità valutazioni dei PO Progetto conti pubblici territoriali e finanza pubblica Attuazione Premialità QCS. Metodi e indirizzi per la valutazione e l'utilizzazione della valutazione Support for project cycle enhancement Supporto e formazione Rete dei nuclei Gruppi di lavoro (GdL) del QCS. Conduzione e committenza di valutazioni su strumenti, piani e programmi Certified assessment of specific sectorial programmes Comunicazione e pubblicazioni. Contributo indirizzi e raccordi attività DPS Metaevaluation of projects Area 1Area 2Area 3Area 4 Methods and studies for public policies Modernization and innovation of administrative action Program EvaluationProject Evaluation Trends and analysis of territorial development policies Support for implementation for DDPservices and for regional and central adm. Coordination of CSF evaluation system Development of guidelines and standards for the assessment of infrastructure and integrated projects Annual report Institutional agreements and FPA, support to design and implementation CSF evaluation and control of evaluation quality of OP Project of territorial public accounts and public finances Implementation of CSF performance reserve. Methods and guidelines for evaluation and use of evaluation Support for project cycle enhancement Support and training to the Network of regional and central evaluation units CSF working groups Conduction and commissioning of evaluation on instruments, plans and programmes Certified assessment of specific sectorial programmes Communication and publicationsInput to DDP guidelines Metaevaluation of projects

Open Days, Rome 3 July Program evaluation Evaluation capacity development –Started with Mezzogiorno & structural funds –Now expanding to whole country & all development program –Huge task: guidelines, groups, observation of practice & processes, common work Evaluations –Slowly growing –Legitimity (who asks for evaluations) –Resources (financial and staff)

Open Days, Rome 3 July Evaluation: where we started from A “continental” public service: No orientation to results Absence of social partners and policy makers Push for evaluation from EU Structural Funds: –No national reinterpretation of general framework –Predominance of EC evaluative needs –Rigid framework Compliance at best purely formal No interactions Low quality of evaluations Little (if any) use of evaluations

Open Days, Rome 3 July Evaluation capacity strategy initially relied on: EU evaluation requirements + New context of policy making –More ambitious development policy –More emphasis on local knowledge –More emphasis on change –More responsibilities to regional governments –Greater importance to administrative capacity

Open Days, Rome 3 July Close links between administration capacity and development policy Development policy requires administrative capacity Evaluation capacity is part of administrative (and political) capacity Evaluation results improve development policies if useful for key actors More than one way to get there

Open Days, Rome 3 July A piecemeal strategy developed over time gradually introducing innovations knowledge from day-to-day work taking into account achievements and disappointments We are still working on it

Open Days, Rome 3 July Some evaluation capacity building activities UVAL worked on Drafting national Guidelines ( ) Efforts to open up the market (2001) Setting up networks and discussion groups (from 2001) Creation of evaluation units ( ) Meta-evaluation activities (from 2002) Self assessment exercises ( ) Utilization of evaluation, evaluative knowledge, and implicit knowledge in discussing strategic lines for next programming period and for nat’l interventions (2005) Orientation to ex-ante evaluation process and products and organization of training and discussion groups (2006)

Open Days, Rome 3 July UVAL role in the evaluation of Ob.1 CSF Organizing the evaluation: Suggested four segments (main contract, environment, cultural resources, macro model) Evaluation questions: Collected MA’s point of view Drafted Terms of Reference Dealt with EC, evaluator, MA Steering Group: Read and commented report drafts and reports Informal dealings, formal meetings and feedback Protected evaluators’ independence

Open Days, Rome 3 July A part of the Network of Italian Evaluation Units Not ALL evaluation units: –Evaluation Units dealing with Development Policies –Within executive branch –Some Ministries and all Regions –There are others: Regional elected assemblies, Universities, other Ministries. Mix of tasks (actually evaluation only a small part) –ex ante analysis, project appraisals, feasibility studies –direct evaluations (only one so far) –management of mid-term evaluations –Various roles in programming activities –Interaction with local authorities and coalitions on projects UVAL plays a proactive role in the Network, but has no formal “central” role.

Open Days, Rome 3 July The National Evaluation System –UVAL – Ministry of Economy and Finance (Coordinator) –INEA – Ministry for agricultural policies –ISFOL – ESF Evaluation Unit – Ministry of Labour –Evaluation Units of MA’s Initially built to deal with evaluation of Objective 1 regions’ strategy and OPs Now on the way to be extended to all Italian Regions’ regional policies

Open Days, Rome 3 July  External : Mid-term evaluation of CSF and OP  Internal:  Evaluation of the performance reserve system  Thematic evaluations (social inclusion, urban development, research)  Self-evaluation Different ways to organise evaluation

Open Days, Rome 3 July EVALUATION OF EVALUATIONS, MID-TERM EVALUATION, UP-DATE Design Early Reports Fieldworks Intermediate Products and in-depth studies Fieldworks Evaluation Questions Organization of evaluation update EC reads Evaluation reports SNV reads reports for lessons and utilization Fieldworks Updating Mid-Term Evaluation Reports SNV reads intermediate evaluation products Working in team with Ob.1 Units (Questionnaire and Meeting in Oct). EVALUTION OF EVALUATIONS MID-TERM EVALUATION OF OP and CSF Mid-Term Evaluation Report Up-dating Mid- term : Strategy of CSF SNV reads preliminary products Work with Ob.1 Units Methodological talks SNV – Evaluators – MA’s – Units SNV makes reports to CSF’s MC