1. Outcome Performance Measures Presentation Derived from Martin & Kettner’s Measuring the Performance of Human Service Programs, Sage, 1996

2. Defining Outcome Measures Results or Accomplishments that are attributable at least in part to a service program (GASB, 1994)

3. Perspectives in Caregiving Problem Approach: I.E. client has housing problem, drug problem etc. is good for diagnosis Quality-of-Life Approach: focuses on end states and attempt to move client toward one of a number of desirable end states.

4. Performance Outcome Measures: 4 Types Numeric Counts –used to measure client conditions, status; behavior Standardized Measures –used to measure client feelings, attitudes, etc. Level of Functioning Scales –measures client, family functioning or condition Client Satisfaction –measures client perceptions

5. Intermediate & Ultimate Outcomes Intermediate Outcome Performance Measures –Numeric Counts –Standardized Measures –LOF Scales –Client Satisfaction Ultimate Outcome Performance Measures –Numeric Counts –Standardized Measures –LOF Scales

6. Selecting Outcome Performance Measures Step 1: Use focus group which includes a representative group of stakeholders to discuss how to assess quality outcomes Step 2: As many outcome performance as possible should be identified Step 3: Group should arrive at a consensus on the best two outcome performance measures for the particular program

7. Cause & Effect In many cases cause & effect relationships between program activities and outcomes cannot be established In other cases cause & effect relationships may exist to the extent that programs contributed, and maybe even contributed significantly to an outcome However, in most cases we avoid the word “cause” and speak of “program contributions” toward a change etc.

8. Social Indicators As Ultimate Outcome Measures Definition: Data that enable evaluative judgements to be made about social problems in a community or state (Miller, 1991) Social Indicators as Ultimate Outcome Measures: Examples –Benchmarking in Oregon (Oregon Options) –Goals 2000 –Healthy People 2000

9. Assessing The Four Types of Outcome Performance Measures: Criteria Utility: percentage of information considered useful Validity: measures what it purports to measure Reliability: measure produces same result repeatedly Precision: capturing incremental changes Feasibility: implementability of measure Costs: relative start up & maintenance costs of measure Unit Cost Reporting: ability to generate cost per outcome data

10. Numeric Counts

11. Numeric Counts:Various Definitions Include demographics and characteristic data related to client flow Nominal measures relating to client flow critical events that reflect, – an undesirable occurrence that an agency is trying to prevent or avoid –a desirable occurrence that the agency is attempting to achieve.

12. Numeric Counts: Examples I&R –intermediate output performance measure (unit of service): one referral –output with quality dimension: one appropriate referral –outcome performance measure intermediate: one client receiving assistance

13. Numeric Counts: Examples Counseling –intermediate output performance measure (unit of service): one hour –output with quality dimension: one hour with counselor of record –outcome performance measure intermediate: one client demonstrating improved behavior ultimate: one client no longer needing service

14. Florida Division of Families & Children Model (Examples) Developmental Disabilities: Behavior Management –one client reported to be exhibiting maladaptive behaviors –one client with a current behavior checklist in his/her file Aging & Adult Services: Adult Day Care Services –one client returned to independent living status –one client prevented from entering a long-term care facility –one client entering a nursing home facility

15. Preference for Numeric Counts: SEA reporting standards promotes use of numeric counts Government Performance & Results Act of 1993 also promotes use of numeric counts Governmental human service programs also prefer numeric counts

16. Assessing Numeric Counts as a Measurement Type UtilityHigh PrecisionLow ValidityLow to Medium ReliabilityFeasibility High CostLow to Medium Unit Cost ReportingHigh

17. Standardized Measures

18. Defining Standardized Measures Standardized measures are validated, reliable and normed pre- post-test measures used to assess quality-of-life changes in clients

19. Standardized Measures: Examples Barthel Index Caregiver Strain Index Index of Clinical Stress Beck Depression Scale Folstein Mini-Mental Assessment

20. Focus Areas for Standardized Measures Population behavior attitude problem intra-personal or interpersonal functioning development personality achievement, knowledge, aptitude services

21. Likert - Scales Most standardized measures use Likert Scale response formats which are typically treated as ordinal or continuous data

22. Translating Standardized Measures Into Numeric Counts: Examples Number of clients demonstrating measurable improvement proportion of clients who demonstrate measurable improvement relative to the number of clients receiving a full complement of services number of clients demonstrating clinical improvement (comparing pre- to post- treatment)

23. Translating Standardized Measures Into Numeric Counts: Examples proportion of clients who demonstrate clinical improvement to the total of clients who completed treatment Proportion of Clients who achieve a target level of improvement proportion of clients who achieved a target level of improvement to the total number of clients who completed treatment

24. An Assessment of Standardized Measures UtilityLow to High ValidityHigh ReliabilityHigh PrecisionMedium to High FeasibilityLow CostHigh Unit Cost Reporting Low

25. Level of Functioning Scales

26. Level of Functioning Scales (LOFs): What are They? Before & after client assessment tools designed for use with a particular human service program that attempts to capture an important dimension of client functioning

27. LOF Characteristics LOFs are typically focused on only one dimension of client functioning Consequently, LOF’s are typically used in combination to assess multiple dimensions of client functioning LOFs are typically (but not always) ranked from very low to very high

28. Principles of Designing an LOF Scale Conceptual Framework: Dimensions of functioning and descriptors anchoring the assessment scale must be based upon a thorough understanding of the program, clients, and the underlying problem Developing Descriptors: –should describe levels of functioning –should discriminate between different function levels –should accurately reflect client behaviors

29. Principles of Designing an LOF Scale Respondent Considerations: –Developing Scales with Client in Mind observe client ask client get information on client from third party

30. Observing Clients Identifying Behavior to be Observed: Involves becoming familiar with scales first, then observing clients Site of Observation: Behaviors change with setting so choose setting or settings Frequency of Observation: Utilize a discrete of settings in which the respondent uses to complete an LOF scale.

31. Observing Clients Avoid Influencing the Observation Situation: Influencing situation make the observed behavior less typical of the client Reliability of Observation: Procedures should be setting for making observations and collecting data, so that different observations are comparable to one another

32. Constructing LOF Scales: Key Steps Step 1: Select the Functions to Be Rated: All functions selected should be expected to change as a consequence of participating in the program Step2: Select the Number of Points on the Scale: Minimum of 3 points (still problematic) to a maximum of 8 points. Usually use 5 point scale Step 3: Write the Descriptors:Should be based on typical, observable verified client behaviors

33. Constructing LOF Scales: Key Steps Step 4: Field Test the LOF Scales: Use on a small scale to obtain experience in observation and coding Step 5: Test Reliability of LOF Scale: When used across many observations of the same client or client group are comparable observation results obtained.

34. Assessment of LOF Scales UtilityLow to High ValidityMedium to High ReliabilityMedium to High PrecisionMedium FeasibilityLow CostHigh Unit Cost ReportingLow

35. Client Satisfaction

36. Client Satisfaction Measures Generates personal attitudes, opinions, feelings and choices Typically scaled from “Not Helpuful at All” to “Very Helpuful” or “Extremely Helpful”

37. Assessment of Client Satisfaction Measures UtilityMedium ValidityLow to Medium ReliabilityMedium PrecisionLow FeasibilityMedium CostLow to High (start-up) Unit Cost ReportingHigh

38. Assessment of Four Types of Outcome Performance Measures

40. Issues in Selecting, Collecting, Reporting & Using Performance Measures

41. Recurring Problems in Federal Grants Failing to relate performance measures to a program’s mission Relying too heavily on existing data Excluding stakeholders from the process Selecting too few quality performance measures

42. Three Key Questions in Reporting Performance Measurement Data How often should performance measurement data be reported? At least annually, semi- annually to quarterly is more functional How much time does it take to collect and aggregate performance measurement data? If performance data is collected regularly and automated then time required ranges from 10 to 20 hours to collect and aggregate a year’s worth of data

43. Three Key Questions in Reporting Performance Measurement Data: Displaying Performance Measurement Data: Inputs: Financial Resources in $; personnel in FTE Outputs: intermediate (time, material); final outputs (service completions) Quality: client satisfaction%; Outputs with quality dimensions Outcomes & Ultimate Outcomes: intermediate outcomes (numeric counts of quality outcomes

44. Three Key Questions in Reporting Performance Measurement Data: Displaying Performance Measurement Data: Cost Efficiency Ratios: Cost per intermediate outputs, & final outputs per FTE; cost per final output Cost Effectiveness Ratios: Cost per intermediate outcome, costs per intermediate & final outcomes per FTE, and cost per ultimate outcome.