How to Assess Student Learning in Arts Partnerships Part I: Focus Groups Revised April 2, 2012 Mary Campbell-Zopf, Ohio Arts Council Craig Dreeszen Ph.D., Dreeszen & Associates

Your Instructors

Our intended learning outcomes Understand key concepts of student learning assessment Discern when interviews and focus groups make sense Design effective focus groups Learn to conduct focus groups

Learning Assessment Assessment of student learning -- a special kind of outcome-based program evaluation Same principles and similar methods as other program evaluations Intended outcomes in arts education are student learning May also be increased teacher and school capacity to teach arts education

Assessment is a Creative Process The art-making process employs mental processes that are critical to thoughtful assessment. Stages of creative process: Exploring Decision-making Art-making—the artistic work takes shape Refining/rehearsing Presenting Cycle begins again  Ohio Arts Council 55

Three Stages of Assessment Define what “good” is – Planning and formative assessment Make the work good – Primarily formative assessment Describe how well it turned out – Summative assessment  Ohio Arts Council 66

Learning Outcomes Learning Outcomes are the knowledge and skills that you expect participants to learn as a result of participating in an educational program or activity. Other key terms: Standards Performance indicators Assessment-embedded instruction  Ohio Arts Council 77

Assessment Process 1.Define intended outcomes 2.Articulate assessment questions 3.Determine data sources and collection methods 4.Select sample or population to query 5.Collect data 6.Analyze data 7.Report findings

Data Sources Existing records Observation (esp. with rubrics) Portfolios Pre- and post-tests Interviews Focus groups Surveys

Focus Groups are Qualitative Research Focused, moderated, & interactive discussion Usually 8-10 people, representative of a specific constituency Discussing opinions & attitudes in response to specific questions 90 minutes to 2 hours

Interviews -- like simple focus groups Set up interviews Develop interview guide of questions Determine sample and schedule Invite to interview Conduct interview Welcome and explain objective of interview Promise anonymity (and keep that trust!) Ask permission to take notes or record Ask questions – Start simple – Move toward more subtle questions Listen well Probe to clarify answers or elicit details Take notes Thank interviewee Summarize notes after interview

Effective Uses of Focus Groups Access constituents & communities Assess needs Explore “whys” behind divergent opinions Clarify an issue for subsequent research Clarify questions and identify opinions to explore with surveys Generate ideas and potential solutions Learn unexpected information Demonstrate respect by asking for advice

Focus Group Advantages Relatively simple and inexpensive Quick results Enables access to specific constituents & communities Believable and usually understandable results Demonstrates willingness to hear constituents

Criticisms and Risks Soft data Moderator influences discussion Participants may not be candid Dominant personalities may influence others Group think is possible Biased interpretation of results possible Risky to generalize from non-representative sample to larger population

Focus Group Steps Define learning outcomes Confirm focus group is appropriate method Develop questions Set location, date & time Recruit moderator Identify and recruit participants Offer incentive (food is good) Conduct focus groups Take notes or record Analyze and report findings

Writing Questions Know your intended learning outcomes What are participants likely to know? Draft short list of short, open- ended questions Shorten list (4 -5 not 10-12) Sequence questions Write probing questions Test and revise …said Alice, "would you tell me, please, which way I ought to go from here?"

Practice Writing Questions Identify an arts education program you’d like to assess. What information do you need? From whom? Write 2-3 group questions for a focus group Try out your questions with colleagues

Leave Some Questions to Policy Makers Focus group participants can only speak of what they know. Collect information about their experiences. But do not ask policy questions: – How shall we allocate resources among competing interests? – How shall we solve a complex problem? – What should be our plan? A messy planning problem

Identify Potential Participants Determine what sector should participate – Invitees share common characteristics (all 8 th grade students or all teachers) – Define target population Select individuals representative of sector

Recruit Participants Letter or – Explain purpose – Use of information – Importance – Present questions – Offer incentive? Phone follow up reminder

Sampling Populations Select representative sample of larger population – Random (random number generator) – Regular (every n th person) – Deliberate (pick representative sample) Sample should look like population Consider statistical significance and margin of error Or question entire small population (the whole class)

Select Location

Recruit Moderator Skilled professionals make a difference – Corporate market research staff (ask your board) – Marketing or social science faculty – Planning, evaluation, or marketing consultants Moderating yourself may work – Testing a program design or policy change – Getting feedback on a draft plan – Asking about need – However, grantees will temper comments to their funder Amateur moderators may skew results – It is easy to hear only what you expect or hope for – Discussions need skillful, light-handed facilitation

Moderating Focus Groups Share refreshments & mingle Open with purpose & process Participants introduce themselves Summarize questions Ask each question – Probe for details and explanations – Encourage discussion – Summarize Be flexible & responsive Draw out quiet people & reign in digressions Thank people at end

Recording Options Direct observation through one- way mirror or participation Audio or video taping – Transcribe into text Take notes – One or two note takers Write up notes

Content Analysis Read notes or transcript Look for and define patterns Assign codes to patterns Mark text with codes Count instances of coded patterns Select typical quotes Write summary – Most frequent responses – Typical quotes Parents’ perception of most important educational benefits for child Most frequently cited benefits (listed in order of most frequent mention, with numbers indicating relative frequency of each idea) Access to excellent teachers (cited by 90) “Our kids have loved it. Beth has been instrumental in fostering and interest in music for them.” Learning good foundation of music skills (50) “…is able to play the instrument and does it with pleasure.” Community of people who love music (39) “…interacting with a group of peers quite different than the peers she engages with at her middles school.”

Report Results Type or dictate notes or transcribe tapes Do content analysis Summarize Draw conclusions Report findings Focus Group Report Seventeen parents and adult students participated in two 90-minute, focus groups. Summary Results: There is much respect and appreciation for high quality teachers. Most feel and value an authentic sense of community. Ensembles are an important part of the learning and community. Most, but not all, appreciate recital performance opportunities.

Use Results Compare focus group results to other data Triangulate Draw conclusions Report to senior staff and/or commissioners Use data to inform decisions

Resources Focus Groups: A Practical Guide for Applied Research, Third Edition, 2000, Richard A. Krueger Focus Groups as Qualitative Research (Qualitative Research Methods), 1997, David L. Morgan

Recommended Practice To really understand focus groups, design one Send one-page focus group design by [date] We provide individual feedback  Program name  Learning outcomes  Potential focus group participants  Focus group questions

Part II Survey Research Steps to develop survey Writing survey questions Online surveys Qualitative and quantitative data analysis Reporting