Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions is a critical phase of the evaluation.

Slides:



Advertisements
Similar presentations
CRITICAL THINKING The Discipline The Skill The Art.
Advertisements

Study Objectives and Questions for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)
What You Will Learn From These Sessions
Chapter 15 Evaluation Recognizing Success. Social Work Evaluation and Research Historically –Paramount to the work of early social work pioneers Currently.
OST Certificate Program Program Evaluation Course
Designing an Effective Evaluation Strategy
Chapter 1: An Overview of Program Evaluation Presenter: Stephanie Peterson.
Clinical Supervision Foundations Module Six Performance Evaluation.
Program Evaluation. Lecture Overview  Program evaluation and program development  Logic of program evaluation (Program theory)  Four-Step Model  Comprehensive.
Chapter 2 Flashcards.
Evaluation.
A brief overview What is program evaluation? How is an evaluation conducted? When should it be used? When can it be used? Used with Permission of: John.
Tailoring Evaluations
PPA 502 – Program Evaluation
PPA 502 – Program Evaluation
Program Management n KSPE 4250 n Ch 2. Vision Statement n A concise statement that describes the ideal state to which the organization aspires. u Include.
PPA 503 – The Public Policy-Making Process
PPA 503 – The Public Policy Making Process
PPA 502 – Program Evaluation Lecture 4a – Qualitative Data Collection.
Evaluation. Practical Evaluation Michael Quinn Patton.
CRITICAL THINKING AND THE NURSING PROCESS
Hartley, Project Management: Integrating Strategy, Operations and Change, 3e Tilde Publishing Chapter 12 Integration Management Practising a common, coordinated.
Outcome Identification and Planning
Getting Started. Decide which type of assessment –Input assessment –Process assessment –Outcomes assessment –Impact assessment.
Health Systems and the Cycle of Health System Reform
Science Inquiry Minds-on Hands-on.
NORMA GOMEZ, MBA, MSN, RN, CNN Unique Challenges for the Nephrology Professional in Managing Change.
How to Develop the Right Research Questions for Program Evaluation
RESEARCH DESIGN.
LeMoyne-Owen College December 16, 2009 Mimi Czarnik, Professor of English and Dean of Humanities Becky Burton, Associate Professor of Biology Alverno College,
Strategic Planning Session David Rudder, Ph.D. Rudder Consultants, LLC. May 17, 2006.
Program Evaluation EDL 832 Jeffrey Oescher, Instructor 6 June 2013.
Semester 2: Lecture 9 Analyzing Qualitative Data: Evaluation Research Prepared by: Dr. Lloyd Waller ©
Creating a Shared Vision Model. What is a Shared Vision Model? A “Shared Vision” model is a collective view of a water resources system developed by managers.
Problem Definition Chapter 7. Chapter Objectives Learn: –The 8 steps of experienced problem solvers –How to collect and analyze information and data.
ISM 5316 Week 3 Learning Objectives You should be able to: u Define and list issues and steps in Project Integration u List and describe the components.
TYPES OF EVALUATION Types of evaluations ask different questions and focus on different purposes. This list is meant to be illustrative rather than exhaustive.
Overview of Chapter The issues of evidence-based medicine reflect the question of how to apply clinical research literature: Why do disease and injury.
Research Seminars in IT in Education (MIT6003) Research Methodology I Dr Jacky Pow.
1 Designing Effective Programs: –Introduction to Program Design Steps –Organizational Strategic Planning –Approaches and Models –Evaluation, scheduling,
McGraw-Hill/Irwin Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. Chapter 3 Identification and Selection of Development Projects.
PPA 502 – Program Evaluation Lecture 2c – Process Evaluation.
Eloise Forster, Ed.D. Foundation for Educational Administration (FEA)
 Welcome, introductions  Conceptualizing the evaluation problem  stakeholder interests  Divergent and convergent processes  Developing evaluation.
AN INTRODUCTION Managing Change in Healthcare IT Implementations Sherrilynne Fuller, Center for Public Health Informatics School of Public Health, University.
Basic Nursing: Foundations of Skills & Concepts Chapter 9
Objectives and Criteria: What are they and how do we use them? Where do they come from? What are the “must-have” criteria? How are market and government.
Evaluation design and implementation Puja Myles
An Expanded Model of Evidence-based Practice in Special Education Randy Keyworth Jack States Ronnie Detrich Wing Institute.
CRITICAL THINKING AND THE NURSING PROCESS Entry Into Professional Nursing NRS 101.
Program Evaluation Principles and Applications PAS 2010.
Evaluating Engagement Judging the outcome above the noise of squeaky wheels Heather Shaw, Department of Sustainability & Environment Jessica Dart, Clear.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 1 Research: An Overview.
Chapter Two Copyright © 2006 McGraw-Hill/Irwin The Marketing Research Process.
UNIT III. A managerial problem can be described as the gap between a given current state of affairs and a future desired state. Problem solving may then.
ICAJ/PAB - Improving Compliance with International Standards on Auditing Planning an audit of financial statements 19 July 2014.
Building Valid, Credible & Appropriately Detailed Simulation Models
Organizations of all types and sizes face a range of risks that can affect the achievement of their objectives. Organization's activities Strategic initiatives.
Pharmacy in Public Health: Community Health Course, date, etc. info.
Cindy Tumbarello, RN, MSN, DHA September 22, 2011.
Critical Thinking or how to learn and know that you know what you know, if you know it Terry C. Norris.
AUDIT STAFF TRAINING WORKSHOP 13 TH – 14 TH NOVEMBER 2014, HILTON HOTEL NAIROBI AUDIT PLANNING 1.
Monitoring and Evaluation Systems for NARS organizations in Papua New Guinea Day 4. Session 10. Evaluation.
Chapter 8 Outcome Identification and Planning Fundamentals of Nursing: Standards & Practices, 2E.
School of Economics Shanghai University
Right-sized Evaluation
The Nursing Process and Pharmacology Jeanelle F. Jimenez RN, BSN, CCRN
Identifying Issues & Formulating Questions
Eloise Forster, Ed.D. Foundation for Educational Administration (FEA)
Presentation transcript:

Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions is a critical phase of the evaluation process The essence of evaluation is generating credible answers to questions about the performance if a social program. Questions must be answerable with the resources available

Evaluation revolves around the hub of carefully crafted evaluation questions Questions need to be meaningful for stakeholders and program decision makers Who will use the the evaluation results? What info do they need? How do they expect to use it? The evaluator’s own analysis of the program is also important

Communication with decision makers and stakeholders Questions are formulated through discourse and negotiation Engaging these groups increases their understanding and appreciation, and makes effective use of the findings when available Evaluators can’t depend solely on their perspectives – they are not experts on evaluation

Role of the Evaluator Framing the questions – the evaluator knows how to analyze a program and focus an evaluation. Raise issues that might be overlooked Identify aspects of program’s operation and outcomes that warrant inquiry Draw out concerns from stakeholders that can be translated into questions that can be answered by evaluation research

Written summary of questions Helps evaluator guide the evaluation design. This is a useful reference for design and selecting research procedures. Can be shared with stakeholders to ensure their concerns are addressed. Safeguards against later misunderstanding of what the evaluation was suppose to accomplish.

The 2 most important topics related to evaluation questions: 1. How to formulate evaluation question that can be addressed using research procedures available to the evaluator 2. How to determine specific questions that the evaluation should focus.

What makes a good evaluation question? Focus the evaluation on the areas of program performance at issue for key decision makers and stakeholders Facilitate design of data collection to provide meaningful info about performance Identify a distinct dimension of program performance that can be credibly assessed

What it means to evaluate something Establishing criteria of merit – on what dimension must the evaluand do well? Constructing standards – how well should should the evaluand perform? Measuring performance and comparing with standards – How well did the evaluand perform? Synthesizing and integrating data into a judgment of merit or worth. What is the merit or worth of the evaluand

……a thought To evaluate anything means to assess the merit or worth of something against criteria and standards.

Dimensions of Program Performance Good evaluation questions must first be reasonable and appropriate The questions must be answerable – specific, concrete, practical, and measurable

Reasonable and Appropriate Questions Work with stakeholders to scale down and focus the questions – Lit Review – social science and social service literature Get acquainted with the program – its structure, activities, roles and tasks of personnel, nature of participants, and assumptions.

Questions must be Answerable Easy to formulate unanswerable questions without realizing it….Does this program enhance family values? Must be possible to identify some evidence or “observables” that can be obtained and will be credible as the basis for an answer. Have measurable performance dimensions stated in unambiguous terms with non-controversial definitions.

Criteria for Program Performance What distinguishes evaluation questions is they have to do with performance and are associated with criteria by which performance can be judged. Performance Criterion (or standard) that apply, as well as the performance dimension at issue. An evaluation that only describes performance, but doesn’t assess it, is not truly evaluation.

Standards by which program performance may be evaluated: Needs/wants of target population Program goals and objectives Professional standards Customary practice, norms Legal requirements Ethical/moral values: social justice, equity

Standards…continued Past performance; historical data Targets set by program managers Expert opinion Pre-intervention baseline levels for target population Conditions expected in the absence of the program Cost or relative cost

Places to look for performance criteria: Professional standards – esp in medical and health programs Prior experience Evaluation and program literature Judgement ratings from stakeholders to establish criterion levels or ranges (low to high performance)

Typical evaluation Questions Need for program services (needs assessment) Program’s conceptulization or design (assessment of program theory) Program operations and service delivery (assessment of program process) Program outcomes (impact assessment) Program cost and efficiency (efficiency assessment)

Order of the five categories or questions The questions draw meaning from the answers to the prior questions. Implementation failure – poor outcomes due to program activities needed to bring about desired improvements did not actually occur. Nutritional status of homeless people did not improve because soup kitchen was rarely open. Theory failure- Program conceptualization and design can’t generate desired outcomes no matter how well implemented – soup kitchen was far away.

Evaluation Hierarchy – Assessment of: 1. Need for the Program 2. Program Design and Theory 3. Program Process and Implementation 4. Program Outcome?Impact 5. Assessment of Program Cost and Efficiency

When developing questions Start at the bottom of the evaluation hierarchy and consider what is known and what needs to be known about the issues Then move up the ladder Logical interdependencies between the levels Premature attention to higher-order evaluation questions can be avoided

Specific Questions Evaluators Should Ask Don’t rely only on input from sponsors or stakeholders – they are familiar with the program and can overlook critical, routine aspects or performance. Evaluator should make an independent analysis of program performance pertinent for investigation

continued Multiple stakeholders = multiple views Set priorities with them and integrate as many relevant concerns as possible If they have constraints, the evaluator must decide whether or not to conduct the evaluation anyway or negotiate to broaden additional perspectives

Essential Strive to ensure stakeholders understand and accept the nature of the eval process, the type of info it will produce, what it might mean if results come out one way or another, and what ambiguities or unanswered questions remain.

Snowball approach Identify and contact a person or group, they then nominate others, who in turn nominate others. The process ends when no new nominations are given.

Topics to discuss with Stakeholders Why the evaluation is needed The program goals and objectives What are the most important questions for the evaluation to answer?

Backward mapping Start with a specification of the desired endpoint and work backward to determine what must be done to get there.

Analysis of Program Assumptions and Theory Construct a conceptual model of how the program is expected to work and the connections presumed between activities and functions and social benefits it is intended to produce. Program theory is the set of assumptions about relationships between strategy and tactics and the social benefits it expects to produce. Program theory is also called program conceptulaization, program plan, blueprint or design.

Logic Model Lays out expected sequence of steps going from program services to client outcomes.

Collating Eval Questions and Setting Priorities Questions tend to cluster around program functions – ex services or outcomes Also around issues – ex need, design, impact When the evaluator prioritizes the questions for a selected program, the next phase is design.

Summary Assess the need for a program Program theory or plan for addressing the need Implementation of the program and associated process Impact or outcome of the program implementation on the social need Efficiency with which the program attains its outcomes