So You Think You’ve Made a Change? Developing Indicators and Selecting Measurement Tools Chad Higgins, Ph.D. Allison Nichols, Ed.D.

Traditional Indicators of Success Achievement on Topic-Specific Tests – Multiple Choice – True/False – Essay Standardized Assessments – Normal Curve

Different Is Not Always Bad! We need to merge traditional classroom assessment with vocational-type assessment and include indicators of whether a participant succeeds or fails to carry out a task or whether organizations make lasting changes. Ways to gather this information may include: Observations Interviews with significant others Focus groups Other qualitative methods.

Why? A gap between education and occupational skill development has been identified by employers. Extension programs should have a lasting effect on clients and communities.

Characteristics of Indicators Indicators should be: – Relevant – Measurable – Based on a scaffold approach – Clear Example: Elderly men and women manage limited financial resources through budgeted spending plan Example of a fuzzy outcome – Senior citizens attain a higher quality of life

Characteristics of Indicators Indicators should be: – Focused on the target audience, i.e age appropriate – Offered to the public – Based on an established norm – Reflect organizational mission and vision – Based upon established research – Specific to a desired outcome

Indicators Should Match the Objectives How will you know participants achieved in the areas listed below? Types of Achievement: Knowledge – Do participants know/understand the subject matter? Reasoning – Can participants use their knowledge to solve problems? Disposition – Have participants developed new attitudes, interests? Performance Skills – Can participants perform the skills they learned over time in their workplace? Products – Are there tangible products? Have conditions changed?

Examples of Indicators Learning: – 90% of the learners will improve their knowledge of subject matter by achieving a higher score on the pre- test than on the post-test. – 90% of the learners will be certified to carry out the skills taught in their workplace environment. Action: – Employers will observe participants successfully using the new skill over a period of three months after coming to the training. – All three employers at the training will incorporate the new procedure or approach into work protocols. Conditions: – Workplace accidents will decrease by 20%. – Families will increase savings by 25%.

Developing Indicators Case Study Activity 1. Divide into two groups. As a group, develop one indicator for each of the three objectives in your case study. 2. Write the indicators on the flip charts.

Case Study – Group 1 Change in Learning (knowledge): – Adolescents learn to distinguish between effective and non- effective decision making. Change in Action (adoption/application): – Pre-teens apply conflict resolution principles in group settings. Change in Conditions (social/economic/organizational) – 4-H clubs adopt procedures that allow teens to be involved in planning and implementing of activities.

Case Study – Group 2 Change in Learning (knowledge, opinions, attitudes): – Parents improve knowledge of ways to handle relationships with their pre-teen and teenaged children. Change in Action (adoption/application of a learned behavior): – Youth and parents practice positive listening skills at family meals. Change in Conditions (social, economic, organizational) – Schools develop new ways to include parents and other community members in educational activities.

Choosing Data Collection Instruments Most important thing to remember!!! – Data collection methods should be selected after determining an evaluation’s purpose and key questions (objectives).

Multiple Methods Because there are weaknesses in every data collection method, a multiple method approach is often best. That includes both quantitative and qualitative methods.

Methods Quantitative – Those methods that gather numerical data. If the data is collected correctly using appropriate sampling, quantitative data may be analyzed with statistical methods and generalized to a larger population. – Examples Questionnaires including: –Needs assessment surveys –Pre and post tests –Post surveys Standardized measure of attainment

Methods Qualitative – Qualitative methods are used to measure meaning. – Words, rather than numbers are used in the analysis. – The results cannot be generalized to a larger population and can only be used to describe the group under study. – Often qualitative information is used to design quantitative instruments. – Examples: Open-ended questions Observations Archival Data/Content analysis Interviews Focus Groups

Things to Consider in Selecting a Method Evaluation questions: What do you want to know? – If the answer to your question is a number or percentage – use a quantitative method. Percentages of people believing or doing certain things? Measurable changes in knowledge or skills? Measurable changes in attitudes or motivations? – If the answer to your question is about what something means to a person or group of people – use a qualitative method. You want to explore how people will use information or skills. You want to know how people feel about an issue. You do not know how people will answer and you want to explore answers.

Things to Consider Evaluator skills Resources Stakeholders preferences Level of acceptable intrusiveness Validity Availability of population Timeliness External requirements

Exercise 1. Divide into groups of 2. Each group will be assigned two data collection methods. 2. Go around the room and view the indicators written in the last exercise. Select an indicator that fits with each of your collection methods. 3. Write the collection method and indicator at the top of the exercise sheet. 4. Write “Advantages” on one side and “Disadvantages on the other side.” 5. Describe your method’s advantages and disadvantages in relationship to the indicators you choose. 6. Report back to the group.

Handout Matrix of Data Collection Methods – Use this form the next time you are designing a evaluation or research protocol.

Where to Go to Find Evaluation Tools List of websites Journal articles Books Contact researchers