1. Rethinking data: Get creative!
Vicki L. Wise
Coordinator of Student Affairs Assessment and Research

2. The potential for meaningful data collection exists all around us
The potential for meaningful data collection exists all around us. We may already have data in our possession that we can use for assessment purposes. The topic of this presentation is rethinking data …

3. These are many choices about how to collect data.
Do you want to collect Qualitative, Quantitative, and/or Mixed Methods data?
Are there existing data that can be used or must new data be collected?
Do we want to collect direct or indirect evidence?
Factors to consider regardless

4. Naturally occurring phenomena Interviews
Journals
Focus groups
Naturally occurring phenomena
Interviews
Audio/
Video files
Observations
Qualitative, Quantitative, Both
Surveys
Pictures
Tests
Reflections
Documents
Performance
Case Studies
Portfolios

5. Qualitative, Quantitative, and/or Mixed Methods
Steps in the Process of Research
Quantitative
Qualitative
Reviewing the literature/previous studies
-Major role
-Minor role
Collecting the data
Standardized instruments: surveys, tests
Predetermined instruments
Numeric data
Large number of individuals
Open-ended questions, Observations/field research, Interviews, Documents, Audiovisual materials
Protocols
Text or image data
Small number of individuals or sites
Analyzing and interpreting the data
-Statistical analysis
-Description of trends, comparison of groups, or relationships among variables
-A comparison of results with predictions and past studies
-Text analysis
-Description, analysis, and thematic development
-The larger meaning
Determining the quality of data
-Reliability
-Validity
-Verification, rather than internal validity
-Transferability, rather than generalizability
-Dependability of the researcher’s account
-Confirmability of findings

6. Are there existing data that can be used or must new data be collected?
First-year orientation
OIRP
Development
Student Success
Think of other groups on campus who are collecting data…can we share?
Alumni, employer, student surveys
Exit interviews with graduates
Graduate follow-up studies
Percentage of students who go onto graduate school
Retention and transfer studies
Job placement statistics
Activities selected or elected by students
Faculty/Student ratios
Percentage of students who study abroad
Enrollment trends
Percentage of students who graduate within five-six years
Diversity of student body

7. Do we want to collect direct or indirect evidence?
students display their knowledge and skills
students or someone else reflect student learning, behavior, attitudes rather than to demonstrate it

8. Regardless of data collection method make sure that the method…
Is linked to values
Measures outcomes that are tied to mission, goals, and objectives
Takes into account any professional standards
Measures expectations
What students need to know, do, or report to assess your intended outcomes
What the program will do to ensure what students will be able to do and to know. Process first, outcomes second.
At a specified level, rather than just being satisfied

9. Regardless of data collection method make sure that the method…
Answers important questions
Collect data you believe will be useful to answering the important questions you have raised
Organize reports around issues, not solely data
Uses reliable and valid measurement
Provides accurate and consistent information (e.g., primary vs secondary source, accuracy, credibility, and trustworthiness) measuring outcomes and appear credible to others
Ensures compliance to the methods (e.g., will participants fill out questionnaires carefully, engage in interviews or focus groups)
Provides enough diversity of information (triangulation, subgroups) to makes decisions about program and participants
Respects human rights/ethics (confidentiality & anonymity, do no harm, appropriate use of data for intended purposes)

10. Regardless of data collection method make sure that the method…
Creates data/information that informs
Stakeholders and decision making
Pedagogy, budgeting, planning, decision making, or policies.
Considers who needs to make decisions with this data and the kinds of evidence for the decisions that need to be made
Considers how information will be reported to convey main points and to inform
Considers available resources
Collected and analyzed in low-cost and practical manner (e.g., using questionnaires, surveys and checklists)
Uses what time and resources are available
Stays in budget and timeline
Considers analysis and interpretation capabilities
Fits into annual responsibilities