1. Answers to your questions
Barbara McNeice-Stallard, Director
Annel Medina Tagarao & John Barkman
Educational Research Assessment Analysts
Research & Institutional Effectiveness
LLRC Retreat, January 2016

2. Ways to evaluate impact: We can help
Focus Groups/Interviews
Surveys
Databases (Banner, Library, Lab)
Over 50 years of experience

3. Focus Groups/Interviews
What was I thinking?
Why?
How do statistician’s minds work?
Why do they read books such as How to Lie with Statistics?

4. Focus Group/Interview Essentials
6-8 people per group (or one-on-one for interviews)
Questions have to be answered with more than yes/no
Record (type like a crazy person; write on flip chart key points)
Guide the discussion, but let deviations happen for awhile
Analysis: Combine themes to re-look at what they are saying (e.g., communications –
what about it? Good or bad?
Suggestions for Change?)
Use of Results

5. Surveys/Questionnaires
How much do you love, love, love your plants? (1=lots but not enough; 5=very much)
Rate the plant arrangement’s color. (1=hideous; 5=magnificent)
Surveys/Questionnaires
Frequency
Opinions
Beliefs
Attitudes
Keep it simple & balanced
Leading Question

6. Survey Essentials makes sense to respondent
Is concrete (e.g., What do you like the most about this course?)
uses important time periods (e.g., within the last week)
uses grade 9 reading level
Use fewest number of words uses few words
avoids biasing (e.g., What do you think of this useful legislation?)
avoids bundling (e.g., using ‘and’ & ‘or’)
avoids negative phrasing (e.g., Why you would not like a poke in the eye?)

7. Data From Data: What is happening?
essential and unique to your program (Luckily…many are already created!)
Library activity
Lab attendance
Banner (Student Information System)
TracDat (outcomes)

8. Database Essentials Quality counts: Garbage in; Garbage out
Ways to summarize (counts, average, %, ???)
Group together Look at details
Standardization: Connect databases for greater insight
Your database impact on students and the college:
Grades
Overdue books
Lab attendance
Student Learning Outcomes
Talking Points:
Definition: This kind of research is done by calculating numbers using the information stored in computer systems.
Example: Banner, the student records system at Mt. SAC, which has information about every enrollment (and a lot more).
What is the system which holds library checkout information?
Getting the information into a computer, instead of stored on paper, changes the research situation from being extremely difficult into being hopefully rather easy.
You have the potential to study a lot of cases (all the enrollments, or every time a book was checked out) but you are limited to only the information which has been recorded on an ongoing basis.
It is best for answering questions about what is happening.
Details:
What is the information which is stored?
What are helpful calculations?
Averages
Frequencies
Percentages
Linking to other databases: This can vastly increase the amount of information which can be accessed, but it requires having a way to link between the databases. At Mt. SAC the best way is usually to record a student’s A number.

9. 3 groups to report out (2 minutes each…seriously!)
Activities (15 minutes)
One per group…
Focus Group
Survey
Database
3 groups to report out (2 minutes each…seriously!)
Petrified Wood

10. Thank you! research@mtsac.edu
Pretend you have your work completed – all of it.
If you get the results you expect, will it actually prove the point?
What if the findings are something you didn’t expect? What would it mean?
How will you use your findings?
If the above makes you uneasy, go back and re-look at your design. Time invested now will save you tons of time (and heartache) later.
Thank you!