Answers to your questions Barbara McNeice-Stallard, Director Annel Medina Tagarao & John Barkman Educational Research Assessment Analysts Research & Institutional Effectiveness LLRC Retreat, January 2016

Ways to evaluate impact: We can help Focus Groups/Interviews Surveys Databases (Banner, Library, Lab) Over 50 years of experience

Focus Groups/Interviews What was I thinking? Why? How do statistician’s minds work? Why do they read books such as How to Lie with Statistics?

Focus Group/Interview Essentials 6-8 people per group (or one-on-one for interviews) Questions have to be answered with more than yes/no Record (type like a crazy person; write on flip chart key points) Guide the discussion, but let deviations happen for awhile Analysis: Combine themes to re-look at what they are saying (e.g., communications – what about it? Good or bad? Suggestions for Change?) Use of Results http://www.mtsac.edu/administration/research/pdf/tips/ResearchTips%20v1n3%20focus%20groups.pdf

Surveys/Questionnaires How much do you love, love, love your plants? (1=lots but not enough; 5=very much) Rate the plant arrangement’s color. (1=hideous; 5=magnificent) Surveys/Questionnaires Frequency Opinions Beliefs Attitudes Keep it simple & balanced Leading Question

Survey Essentials makes sense to respondent Is concrete (e.g., What do you like the most about this course?) uses important time periods (e.g., within the last week) uses grade 9 reading level Use fewest number of words uses few words avoids biasing (e.g., What do you think of this useful legislation?) avoids bundling (e.g., using ‘and’ & ‘or’) avoids negative phrasing (e.g., Why you would not like a poke in the eye?) http://www.mtsac.edu/administration/research/pdf/tips/ResearchTips%20v1n1%20designing%20surveys.pdf

Data From Data: What is happening? essential and unique to your program (Luckily…many are already created!) Library activity Lab attendance Banner (Student Information System) TracDat (outcomes)

Database Essentials Quality counts: Garbage in; Garbage out Ways to summarize (counts, average, %, ???) Group together Look at details Standardization: Connect databases for greater insight Your database impact on students and the college: Grades Overdue books Lab attendance Student Learning Outcomes Talking Points: Definition: This kind of research is done by calculating numbers using the information stored in computer systems. Example: Banner, the student records system at Mt. SAC, which has information about every enrollment (and a lot more). What is the system which holds library checkout information? Getting the information into a computer, instead of stored on paper, changes the research situation from being extremely difficult into being hopefully rather easy. You have the potential to study a lot of cases (all the enrollments, or every time a book was checked out) but you are limited to only the information which has been recorded on an ongoing basis. It is best for answering questions about what is happening. Details: What is the information which is stored? What are helpful calculations? Averages Frequencies Percentages Linking to other databases: This can vastly increase the amount of information which can be accessed, but it requires having a way to link between the databases. At Mt. SAC the best way is usually to record a student’s A number. http://www.mtsac.edu/administration/research/pdf/tips/Keeping_a_Clean_Local_Database.pdf

3 groups to report out (2 minutes each…seriously!) Activities (15 minutes) One per group… Focus Group Survey Database 3 groups to report out (2 minutes each…seriously!) Petrified Wood

Thank you! research@mtsac.edu Pretend you have your work completed – all of it. If you get the results you expect, will it actually prove the point? What if the findings are something you didn’t expect? What would it mean? How will you use your findings? If the above makes you uneasy, go back and re-look at your design. Time invested now will save you tons of time (and heartache) later. Thank you! research@mtsac.edu