Simple Survey Resources: Templates, Tabulation & Impact Jeff Buckley, Jenna Daniel, & Casey Mull

Overview Situation Need Being Met Audience (Agents & Associates) Basic Overview of the Training How Can This Be Utilized in Your State?

Situation As a result of shrinking state budgets, the need to effectively document program impact is more critical than ever. Many states do not have a full time Evaluation Specialist. County Agents and State Specialists must develop their own evaluation instruments. Many County Agents and State Specialists may not feel qualified or confident enough to develop their own instruments and/or to tabulate and report data.

Need Being Met A 4-H program evaluation training was created Implemented by Georgia 4-H State Specialists & CAES Faculty Piloted at the 2012 Winter School – 4-H Agents & Associates

Simple Survey Training Training Format: – 90 minute session – Interactive – 4-H Agents and Associates – Participants created their own program evaluation

Simple Survey Training Why is it Important to Gather Data? – Who is your audience? – Process Evaluation – Internal Use – Outcome Evaluation – External Audience How will you share the results? – Impact Statement – News Article – Journal Article – Other

Simple Survey Training What Can a Survey Measure? – Knowledge – Attitude – Behavior Designing Measurable Objectives – Refer to the handout “Verbs for Writing Measurable Objectives.” – Measurable objectives will translate into more effective constructs and questions.

Simple Survey Training Consider… – What you want your participants to know. – How you want their attitude to change. – What sort of behavior changes you want to see. Creating A Construct – Pick two outcomes/constructs and develop three “questions” each, OR – Pick three outcomes/constructs and develop two “questions” each. – (The template has space for six “questions”.)

Simple Survey Training Creating A Construct – The food in the cafeteria was hot. – The dining area was clean and comfortable. – There were a variety of fruits and vegetables. – Beverage selections included healthy options. – Vegetables were not over-cooked. – The servers were on task and pleasant.

Simple Survey Training Likert Style Questions – These templates are designed for use with Likert style questions. – Pick the rating scale that works best for your objectives/program. – This is the most challenging part of developing your survey. Allow time. Ask for feedback.

Simple Survey Training Likert Style Questions – Consist of a statement and a rating scale. – Need to conform to one rating scale. – Measurable verbs that fit the appropriate cognitive domain. – One data point per question. – Is what you’re measuring important to the intended audience of the evaluation?

Simple Survey Training Instrument Templates – Traditional – Retrospective Postthenpre – 6 quantitative items – 1 qualitative item – Feedback on the program

Simple Survey Training Tabulation – Excel Spreadsheet – Directions – Worksheet for both instrument templates – “Locked” format

Now You’re Ready to Gather Data!

Results Presentations Conducted – Galaxy – Georgia 4-H Winter Conference – Georgia Association of 4-H Agents Conference – Southern Region Volunteer Conference

Results Impact – The percentage of program participants who stated that they could… – develop measurable objectives increased from 60% before to 100% after the program. – design evaluation instruments for their programs increased from 69% before to 91% after. – are likely to develop their own evaluation tools in the future increased from 46% to 89/%. – A domain analysis of the qualitative revealed the following themes… – Participants were very glad to have the templates. – Participants plan to use the tools and knowledge they gained to gather data. – Participants plan to use the data they collect to develop higher quality impact statements.

Results Extension Agent Quotes & Impact Statements – With a mean score of 3.1 (SD=.994), overall respondents from Question 5 indicated they "agreed" they are more likely to get involved in their community as a result of the GPK Leadership Adventure weekend. – Retrospective post then pre: [In Question 2], 84.6% of respondents indicated an increase in their confidence concerning their ability to “design or construct a map of the distribution of major world languages”.

Simple Surveys in Florida 4-H Tabulation of Data – Dropping the data in the sheets is so simple and saves a ton of time. It is much easier than manually tabulating learning gains – Data is easy to interpret from the sheets How Data Has Been Used – Data has been used with stake holders to secure program support and has been used in annual reports of accomplishment. – Data will be used in presentations at professional conferences to show outcomes and projected impacts of programming. 5.

Training Evaluation Training Program Evaluation Exercise – Confidence in ability – Knowledge gain – Training evaluation

How Can You Utilize this Program? Resources: – State Meetings Webinars New Agent Training

Questions? For more information contact: University of Georgia, State 4-H Office Jeff Buckley, Jenna Daniel, Casey Mull, State 4-H Office –