Evaluation Capacity Building in a National Context: View from the Academy in Canada Susan Phillips Robert P. Shepherd Professor & DirectorAssistant Professor School of Public Policy and Administration Presentation to the AEA Conference November 12, 2010 San Antonio, Texas

What is Driving Evaluation Training and Research in Canada? Evaluation is driven by two main purposes: –Evaluation for learning (improvement, effectiveness). –Evaluation for accountability (satisfying government demands). Most evaluation conducted in Canada is for government purposes - accountability Evaluation is almost entirely supply-driven (government has the supply of programs to evaluate, and determines the conditions for demand) Government needs evaluators who understand their priorities, but universities teach research methods 2

So, what is driving federal evaluation and university training programs? The OAG concluded that “departmental evaluations covered a relatively low proportion of its program expenses – between 5 and 13 percent annually… and departments do not regularly identify and address weaknesses in effectiveness evaluation.” Timely results from evaluation studies – the new Federal Evaluation Policy reinforces a very short term focus by linking evaluation to “strategic reviews.” Less technical (meaning less evaluation), more monitoring of program performance and immediate results Linking evaluation with expenditure management decisions – evaluation supports “corporate” priorities, not better programs Evaluators that understand government priorities and demands, not necessarily evaluation practices and methods 3

Goals for Canadian Universities: More skilled method evaluators –Who know different contexts of evaluation –Can ‘scale up’: from activity to policy evaluation – effectiveness! Better consumers of evaluation (public managers) Contribute to professionalization of the field –Better credentials –Capacity building Advanced research in evaluation 4

Evaluation Programming in Canadian Universities: The Consortium EnglishBilingualFrenchDedicated Programs Alberta Carleton Dalhousie Fraser Valley Queen’s Royal Rhodes Saskatchewan Victoria Waterloo U. Of New Brunswick Ottawa ENAP U de Montréal Laval Public Admin 6 Education 5 Health 4 Psychology 5 New Consortium of Universities for Evaluation Education 5

Evaluation Education : Key Criteria Relevant to Users – Positivist approaches that emphasize absolute precision and take too much time have been rejected since 1977 – sometimes “good enough” on attribution is ok Rigorous- right level of challenge that balances concerns for methods with user needs – realistic! Reach – Accessible and appeals to right people –Distance –Cost Mix of credentials & programming (degree & non- degree) with the possibility of laddering into degrees Appeal to mix of students: practitioners & new entrants 6

Canada has a new Credential The Canadian Evaluation Society (CES) now has a CE designation Many universities are recognized trainers for new entrants to the profession Heads of Evaluation will be required to be credentialed as a job requirement Consultants will soon need the credential in order to bid on federal evaluation projects The Consortium works closely with the CES and the Centre for Excellence in Evaluation (TBS) 7 CE

Carleton Experience School of Public Policy & Administration is the oldest and largest in Canada (established in 1953) Masters concentrations: –Policy Analysis –Public Management (including evaluation) –Innovation, Science & Environment –International & Development Admits about 65 Master’s students and 12 Doctoral students per year 8

The Carleton Experience Centre for Policy & Program Assessment (CPPA) celebrated 30 years in 2009 –Evaluation training & capacity building (e.g. UNICEF, UNIFEM, various federal departments, City of Ottawa) –Conducts evaluations –Publishes How Ottawa Spends IPDET is 10 years old – the international connection Multidisciplinary faculty: several do evaluations but do not regard themselves as “evaluators” 9

Graduate Diploma in Policy and Program Evaluation (DPPE) Established 2007 Intent is to provide advanced education in both the methods and management of evaluation DPPE consists of 6 Masters level courses: –4 Required: Policy & Program Evaluation, 2 Research Methods & Capstone –2 electives: (e.g. Cost/Benefit & Strategic Policy Evaluation) Can be completed in 14 months while working full-time Integrated with IPDET: can get credit for up to 3 of 6 DPPE courses DPPE can be applied to Masters in Public Administration Admits students each year 10

What is Missing for Evaluation in Canada Accessibility –Lack of distance education but this is changing –Cost for international students is very high Capacity building with universities outside Canada Undeveloped partnerships with CES/CEE, others to develop expertise in specialized areas (e.g., development, gender, cultural, social) Co-ops /Internships / placements for students Evaluation research –Compared to the USA, Canadian universities have no ‘big names;‘ this is less problematic than lack of programming & research in evaluation BUT, a key contribution for Canadian universities is building a culture of evaluation (demand), not just monitoring! 11

The Dream Sophisticated hybrid/distance learning & even closer integration with IPDET and other international programs Education for the next generation – of evaluators and policy makers/managers who understand effectiveness evaluation –Strengthen evaluation focus – move to dedicated degrees –Develop student exchanges and internships Expanded research – with a focus on evaluation in the international context –Grow the pool of researchers –Convert faculty to evaluation research –Research Chair in Evaluation Develop specialty professional development courses (through partnerships – both short and long) More Master’s and PhD level programs 12

Getting There Takes... Resources –development of distance education courses –Research Chair in Evaluation –Other faculty and associates (starting to build relationships) Partnerships –The Consortium (this is our best means to move forward) –IPDET –CES, AEA and other associations –International agencies –Other universities domestic and international –Consulting firms: provide internship opportunities –Researchers: need to incorporate knowledge and experience Time & patience, but not too much: Now is the Time 13

Contact Us: Susan Phillips Director School of Public Policy & Administration Robert P. Shepherd Supervisor, DPPE Program Chairperson, CUEE 14