PPA 502 – Program Evaluation Lecture 2c – Process Evaluation

Introduction  You must figure out exactly what the program theory is—that is, you must articulate the program theory; you must make it explicit, so that it can be examined and evaluated.  Typically, a program theory will be implicit, at best. That’s why the evaluator must understand how the program is intended to operate.  Program theory (like needs assessment) is very important, because it is foundational for later types of evaluation in the RFL model (i.e., implementation assessment, impact assessment, and efficiency assessment).

Introduction Overview of Program Theory Target population Service Arena Program Service Utilization Plan Program's Organizational Plan Impact Theory Target interaction with delivery system Program - target service interactions Program facilities, personnel, activities Proximal Outcomes Distal Outcomes

Program Theory  A social program centers on the transactions that take place between a program’s operations and the population it serves. –Program impact theory. –Service utilization plan. –Program organizational plan.

Program Theory  Program impact theory. –Assumptions about the change process started by the program and the improved conditions expected to result.

Program Theory  Service utilization plan. –The program’s assumptions and expectations about how to reach the target population, provide and sequence service contacts, and conclude the relationship when services are no longer needed or appropriate.

Program Theory  Program’s organizational plan. –Program resources, personnel, administration, and general organization. –Resources, organization, and activities will produce a viable organization that will successfully implement the service utilization plan.  Process theory. –The service utilization plan and the organizational plan combined.

Process Evaluation  Process evaluation assesses the success or failure of the program theory.

Three Major Questions of Process Evaluation  What is the program intended to be? –Methods to develop and specify program components  What is delivered in reality? –Methods for measuring program implementation  Why are there gaps between program plans and program delivery? –Assessing influences on the variability of implementation.

Chain of Events Logic for Process and Impact Evaluation

Process evaluation to develop and specify program components  A major function of process evaluation is using data to help design interventions to clarify and obtain agreement on what is the intended program.

Types of program components  Intended recipients. –Background characteristics appropriate for program. –Eligibility requirements. –Recruitment mechanisms. –Selection process.  Intended context. –Types of agencies that will deliver the program, and their characteristics. –Community context.

Types of program components  Intended delivery. –Activities: Who does what? With whom? –Staffing: What types, backgrounds, and skills will be used? –Materials needed for delivery. –Information and information systems.  Intended scope of program.

Criteria for measurable program components  Specify activities as behaviors that can be observed rather than as goals or objectives.  Ensure that each component is separate and distinguishable from other components so that each one can be separately measured.  Explicitly link each component to its underlying theoretical rationale.  Include all activities and materials intended for use in the intervention.  Identify the aspects of the intervention that are intended to be adapted to the setting, and those that are intended to be delivered as designed.

Techniques for Program Specification  Formative Evaluation –Data collected from pilot situations and recipients while developing an intervention to obtain feedback about the feasibility of proposed activities and their fit with intended settings and recipients.  Evaluability Assessment –A set of systematic processes for developing the underlying program theory and clarifying intended uses of data before initiating full-scale evaluation.  Use of Theory to Aid Program Specification –Applying normative and causal theories relevant to the content area and using data to elucidate the underlying processes.

Data collection methods for formative evaluation  Focus groups  Observation  Open-ended interviews  Ethnographic analysis  Message or form analysis  Expert judgment  Equipment trial

Process evaluation methods for measuring program implementation  The program components are the basis for selecting or developing instruments to measure two key aspects of program delivery: the extent of implementation (number and quality of components delivered) and the scope of implementation (number of recipients reached and their characteristics).

Uses of implementation data  Monitor current activities to identify problems in implementation, then improve service delivery.  Measure variability in program delivery for later statistical analyses of program impacts.  Use as dependent variables in assessing why delivery is or is not carried out as intended.

What to measure  Number of participants reached and their characteristics.  The number of program activity components and their characteristics.

How to measure implementation  Use of technical equipment.  Indirect unobtrusive measures.  Direct observation.  Activity or participation log.  Organizational records.  Written questionnaires.  Telephone or in-person interviews.  Case studies.

Problems in implementation measurement  Single measures.  Reliability and validity.  Representativeness.  Implementation should be explicitly linked to outcomes for impact assessment.

Assessing influences on the variability of implementation  The implemented program is likely to vary from original intentions. –Why does it occur? –How does it effect program effectiveness? –What might be done to create full implementation?

Macro-implementation versus micro-implementation  Macro-implementation: large scale programs. Focus on intergovernmental management. –Decision points, competing priorities, interests and resources of the various actors, and diverse strategies for change.  Micro-implementation: the extent of compatibility between the pre-existing organization and the new program requirements.

Influences on micro- implementation  Organization as a whole –Types of decision-making processes used – centralized or participatory. –Procedures and priorities for allocating resources. –The supportiveness of an organization’s overall culture toward programs of the type implemented. –The types of pressures emanating from the organization’s environment of beneficiaries, supporters, competitors, and regulators.

Influences on micro- implementation  Structures and processes of an organization’s work units. –Expectations and performance feedback from the supervisors of each unit. –Technologies in use and their fit to the new program. –Standard operating procedures used to simplify and regulate work flow. –Use of time and availability of time to learn activities for the new program. –Social norms governing ways of working that have developed within a unit. –Communication processes both within and between units.

Influences on micro- implementation  The actions of individual deliverers (“street-level bureaucrats”). –Their own capabilities or skills in using the innovation. –Extent and types of training they are given for delivering the new program. –Their concerns about how it affects them personally, such as its possible effects on their future employment and advancement. –External incentives and internal motivations to learn and to deliver the new program.

Methods for Data Collection and Analysis  Case Studies  Systematic surveys of organizational members.

Suggested roles for process evaluation  The usefulness of evaluation for improving program management and delivery will be substantially increased by more emphasis on data about program processes.  Evaluations of the impact of program interventions should always include measures of the extent of program delivery.

Suggested roles for process evaluation (contd.)  Understanding of program impacts, whether or not the desired change occurred, will be greatly strengthened by process evaluation data.  The more that variation in program delivery is expected among multiple sites, the greater the need for process evaluation.  The larger the scale of an evaluation study in number of sites and participants, the greater the need to measure the extent and processes of intervention delivery.