How to Assess Student Learning in Arts Partnerships Part II: Survey Research Revised April 2, 2012 Mary Campbell-Zopf, Ohio Arts Council Craig Dreeszen Ph.D., Dreeszen & Associates

Your Instructors

Our intended learning outcomes Understand the use and limitations of written assessment instruments Write effective survey questions Understand survey distribution options Interpret survey data – Qualitative data analysis – Quantitative data analysis Report survey findings

Written Assessment Instruments Evaluation forms Tests – Pre- and post-tests – Standardized tests Surveys

Using Written Assessments Good reasons to use: Efficient May reach large samples Test qualitative conclusions from small samples Provide quantitative data Enables reticent people to participate May not be appropriate if: Uncertain what to ask Questions are ambiguous Questions are complex Respondents not literate

Two Sequencing Options Start with interviews or focus groups: Test questions Learn likely responses Explore conclusions Create fixed-response questions Distribute to larger sample Start with survey: Explore ambiguous survey findings in interviews or focus groups Investigate reasons for survey findings

Survey Steps 8/12/2015 Decide what data is needed Design questionnaire Select sample Acquire lists Pretest Distribute and remind Analyze and report

Determine Needed Data Learning outcomes – To what extent were did participants learn what was intended – Unintended outcomes Program satisfaction – What liked best – What needs improvement Recommendations Marketing advice Advocacy evidence

Write Questions Open-ended questions Short answer Long answer Prompted Fixed-response questions Yes/No Rating Scale Choice (one or multiple) Matrix rated Ranked Ask basic demographic questions too

Review Sampling Select representative sample of larger population Or survey entire population Acquire mailing or lists

Open-ended Questions

Fixed-Response Questions [Replace sample with OAC AP grantee examples]

Practice Writing Questions Try writing an open-ended question Then, write a fixed- response question and response choices

Paper and Online Surveys Mailed surveys – Envelopes – Address labels – Postage – Cover letter – Survey – Return envelope and postage Handout surveys Online surveys SurveyMonkey.com Zoomerang.com – invitation – Survey – May provide links to survey results and organization website – Survey closed notice

Online Survey Features Screening questions Skip logic Quick response Inexpensive Track responses Send reminders to unresponsive addresses

Test Survey Send early draft survey to partners Revise Send next draft to small sample of your population Ask for feedback Look for ambiguity or misunderstanding Revise final survey

Distribute Surveys Paper surveys Address and mail Remind (newsletter, announcements…) Handout surveys Distribute and collect Online surveys Time right invitation with link to survey URL Web site links Social media links Track responses Send reminders

Statistical Significance Response Rate Number of responses divided by number of surveys distributed 90 responses 200 surveys sent = 45% response rate Deduct undeliverable surveys Margin of error When sampling populations, larger samples have smaller margin of error Use online calculator If you survey 200 students and 90 respond, margin of error = 7.7 If 120 respond, margin of error is 5.7 Some apparent results may be due to chance – not statistically significant No margin of error if entire population sampled, e.g., all participants in your program

Analysis of data 8/12/2015 Reduce large amount of data Display it in notes, tables, or charts Make sense of it – Statistical analysis of numbers – Content analysis of words

Making sense of words 8/12/2015 For each question, read all answers Look for patterns and clusters Code similar statements Count clusters of similar statements Select typical quotes Write summary Do the same for next question

Making sense of numbers 8/12/2015 Count Average Note patterns and clusters Note most frequent responses Cross tabulate to compare variables Check statistical significance Write summary

Report Results with Text Executive summary Interpret results Key-point summary Headings Explanation Data Quotes Appendix with detailed data and methods “Findings: The Arts Partnership Program works as intended. The Arts Partnership program furthers the OAC strategic plan, especially Goal 3, to “Help citizens of all ages learn and thrive through the arts.” Quantitative evidence cited Just over half of AP grantees (55%) cited quantitative evidence to support their evaluation results in the final report.”

Display Results with Graphics Data tables Bar charts Histograms Pie charts Line charts Scatter plots

Report Results Report intended learning objectives and actual learning outcomes Summarize activities Suggest best practices and needed improvements Acknowledge funders and partners Provide back-up data Sample OLGA report form

Use Results Draw conclusions Report to stakeholders Use data to improve programs and account to funders authorizers