Patricia Rogers, RMIT University, Melbourne Part 3 of 8 AEA Coffee Break Webinars 2013 FRAME what is to be evaluated.

Slides:

Advertisements

Similar presentations

Program Evaluation Alternative Approaches and Practical Guidelines

Advertisements

Narrative Synthesis in Systematic Reviews

Patricia Rogers April 23, 2012 Rome IMPROVING EVALUATION BY SHARING INFORMATION ABOUT METHODS.

Introduction to Monitoring and Evaluation

Study Objectives and Questions for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)

Protocol Development.

What Happened? In your group, discuss what happened in your district regarding SB 1 after our last meeting. Each group will share (1)one positive outcome.

Workshop 501 and 505 Review barriers to communication

 Systematic determination of the quality or value of something (Scriven, 1991)  What can we evaluate?  Projects, programs, or organizations  Personnel.

Kerry Bruce - Pact MANAGE your evaluation Part 8 of 8 AEA Coffee Break Webinars 2013.

MOOCs and the Quality Code Ian G. Giles PFHEA Medical Education

Getting on the same page… Creating a common language to use today.

The Accountant’s Role in the Organization

S519: Evaluation of Information Systems Understanding Evaluation Ch1+2.

PPA 503 – The Public Policy Making Process

Designing and Developing Interactive Multimedia EDCI 663 Educational Technology Purdue University.

Technical Writing Function. The purpose of having guidelines is to make the document more readable. Standard guidelines govern – Format – page layout,

Types of Evaluation.

Chapter 2 A Strategy for the Appraisal of Public Sector Investments.

Simon Hearn, ODI, London Part 2 of 8 AEA Coffee Break Webinars 2013 DEFINE what is to be evaluated.

The Accountant’s Role in the Organization © 2009 Pearson Prentice Hall. All rights reserved.

La Naturaleza.  The generic term for the five-phase instructional design model consisting of Analysis, Design, Development, Implementation, and Evaluation.

The phases of research Dimitra Hartas. The phases of research Identify a research topic Formulate the research questions (rationale) Review relevant studies.

Results-Based Management

Simon Hearn, ODI, London Part 7 of 8 AEA Coffee Break Webinars 2013 REPORT and SUPPORT USE of findings.

Irene Guijt, Learning by Design Part 4 of 8 AEA Coffee Break Webinars 2013 DESCRIBE what happened.

Key concepts & principles of assessment. What is Assessment ? Assessment for Learning is the process of seeking and interpreting evidence for use by learners.

Too expensive Too complicated Too time consuming.

Accountability in Health Promotion: Sharing Lessons Learned Management and Program Services Directorate Population and Public Health Branch Health Canada.

Elizabeth Godfrey 1.  Periodic assessment of results Appropriateness, effectiveness, efficiency, impact, sustainability  Identifies intended and unintended.

Comp 20 - Training & Instructional Design Unit 6 - Assessment This material was developed by Columbia University, funded by the Department of Health and.

Module 3: Unit 1, Session 3 MODULE 3: ASSESSMENT Adolescent Literacy – Professional Development Unit 1, Session 3.

Setting the Stage: Workshop Framing and Crosscutting Issues Simon Hearn, ODI Evaluation Methods for Large-Scale, Complex, Multi- National Global Health.

‘CHILD’S PLAY’ THE EARLY YEARS LEARNING FRAMEWORK Child Studies 1 – Group Task.

Kerry Bruce - Pact MANAGE your evaluation Part 8 of 8 AEA Coffee Break Webinars 2013.

1 Unit 1 Information for management. 2 Introduction Decision-making is the primary role of the management function. The manager’s decision will depend.

Assessment in Education Patricia O’Sullivan Office of Educational Development UAMS.

Teaching Today: An Introduction to Education 8th edition

Rogerian Argument Taken from the principles of psychologist Carl Rogers.

USEFULNESS IN ASSESSMENT Prepared by Vera Novikova and Tatyana Shkuratova.

McGraw-Hill/Irwin © 2005 The McGraw-Hill Companies, Inc. All rights reserved Performance Appraisals Chapter 11.

|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

Employee Engagement (5EEG)

The selection of appropriate assessment methods in a course is influenced by many factors: the intended learning outcomes, the discipline and related professional.

Assessment Design. Four Professional Learning Modules 1.Unpacking the AC achievement standards 2.Validity and reliability of assessments 3. Confirming.

Cost Accounting Dr. Baldwin University of Alabama in Huntsville Spring 2010.

CES 20th Annual Conference Keun-bok Kang (CNU) & Chan-goo Yi (ETRI) ; Taejon, Korea A DESIGN OF THE METAEVALUATION MODEL A DESIGN OF THE METAEVALUATION.

Draft NONIE guidance March 2009 Discussant comments Patricia Rogers.

Evaluation design and implementation Puja Myles

Program Evaluation Overview. Definitions of Program Evaluation systematic collection of information abut the activities, characteristics, and outcome.

Scanner Slip-Up Case: Planning and Drafting. What are the choices? Complete the Project Planning Sheet for this case. This is just an initial plan. There.

Reading Take-Aways 1.Which of the four C’s do you think is most critical for student preparation for PJL? Why? 2.How does a driving question facilitate.

1 Classroom Assessment Compiled by Linda Blocker.

What is Impact Evaluation … and How Do We Use It? Deon Filmer Development Research Group, The World Bank Evidence-Based Decision-Making in Education Workshop.

.  Evaluators are not only faced with methodological challenges but also ethical challenges on a daily basis.

The development of self directed support systems in Glasgow Glasgow City Council.

Chapter 6 Assessing Science Learning Updated Spring 2012 – D. Fulton.

P3 Business Analysis. 2 Section F: Project Management F1.The nature of projects F2. Building the Business Case F4. Planning,monitoring and controlling.

7 common challenges in using theory of change - and how to address them Professor Patricia Rogers BetterEvaluation Royal Melbourne Institute of Technology,

Main strand session 17 Session Seventeen Measuring Learning 2: Assessment evidence Jim Rogers.

The Achievement Chart Mathematics Grades The primary purpose of assessment and evaluation is to improve student learning.

Cluster Host Preparation Meeting Autumn Term 1a Overview and Action Planning Judith Carter Senior Adviser Complex Needs/Vulnerable Learners

The Accountant’s Role in the Organization

Draft NONIE guidance March 2009

SMART Objectives Are in addition to day-to-day accountabilities

Business Case 24 Jan 2017 How many of you have ever created a business case? How was the process?

The Accountant’s Role in the Organization

The Accountant’s Role in the Organization

The Accountant’s Role in the Organization

Draft NONIE guidance March 2009

Presentation transcript:

Patricia Rogers, RMIT University, Melbourne Part 3 of 8 AEA Coffee Break Webinars 2013 FRAME what is to be evaluated

The Rainbow Framework

Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Make Decision

Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake Decision

Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake DecisionDesign Evaluation

Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake DecisionFrame EvaluationDesign Evaluation

Four evaluation tasks in FRAMING Identify primary intended users Decide purpose(s) (intended uses) Specify key evaluation questions Determine what ‘success’ looks like

Identify primary intended users 1.

Possible primary intended users Image source: IN 157s01 – Curt Carnemark/ World Bank

Decide purpose(s)(intended uses) 2.

Formative – improve it Summative – continue or stop it Broader evidence base Purposes (intended uses) Image source: CK-CO138 - Charlotte Kesl / World Bank Lobby and advocate

Purposes (intended uses) Build trust and legitimacy Accountability Provide voice

Interconnection between intended users and intended uses Identify primary intended users Decide purpose(s) (intended uses) FRAME

Specify the key evaluation questions 3.

Descriptive: How many children attend? What learning tools are used? Has learning improved? Descriptive: How many children attend? What learning tools are used? Has learning improved? Key evaluation questions Image source: ML030S09 - Curt Carnemark / World Bank

Causal: Has the program contributed to improved learning? Causal: Has the program contributed to improved learning? Key evaluation questions

Synthesis: Has the program been a success? Is it Value For Money compared to alternatives? Synthesis: Has the program been a success? Is it Value For Money compared to alternatives? Key evaluation questions

Action: How can the program be improved? Should it continue? Action: How can the program be improved? Should it continue? Key evaluation questions

MANAGE DEFINE FRAME DESCRIBE UNDERSTAND CAUSES SYNTHESIZE REPORT & SUPPORT USE Descriptive Questions- What were the activities, changes, context? Causal questions –What caused or contributed to the identified changes? Synthesis questions – Overall was it good? Value for money? Action questions- What should we do? Options for answering different types of questions

Determine what success looks like 4.

Processes: Are students fairly treated? What does success look like? Image source: LQ-CN7499 World Bank

What does success look like? Outcomes: Do students learn to read?

What does success look like? Distribution of costs and benefits: Who benefits from the program? Whose needs are met? Who is disadvantaged by it?

What does success look like? Criteria and standards: Reading levels – better than before, or better than national average, or national benchmark?

You can read about this case here : /outcome_harvesting_bionet

Patricia Rogers RMIT University Further resources