Rethinking data: Get creative! Vicki L. Wise Coordinator of Student Affairs Assessment and Research

The potential for meaningful data collection exists all around us The potential for meaningful data collection exists all around us. We may already have data in our possession that we can use for assessment purposes. The topic of this presentation is rethinking data …

These are many choices about how to collect data. Do you want to collect Qualitative, Quantitative, and/or Mixed Methods data? Are there existing data that can be used or must new data be collected? Do we want to collect direct or indirect evidence? Factors to consider regardless

Naturally occurring phenomena Interviews Journals Focus groups Naturally occurring phenomena Interviews Audio/ Video files Observations Qualitative, Quantitative, Both Surveys Pictures Tests Reflections Documents Performance Case Studies Portfolios

Qualitative, Quantitative, and/or Mixed Methods Steps in the Process of Research Quantitative Qualitative Reviewing the literature/previous studies -Major role -Minor role Collecting the data Standardized instruments: surveys, tests Predetermined instruments Numeric data Large number of individuals Open-ended questions, Observations/field research, Interviews, Documents, Audiovisual materials Protocols Text or image data Small number of individuals or sites Analyzing and interpreting the data -Statistical analysis -Description of trends, comparison of groups, or relationships among variables -A comparison of results with predictions and past studies -Text analysis -Description, analysis, and thematic development -The larger meaning Determining the quality of data -Reliability -Validity -Verification, rather than internal validity -Transferability, rather than generalizability -Dependability of the researcher’s account -Confirmability of findings

Are there existing data that can be used or must new data be collected? First-year orientation OIRP Development Student Success Think of other groups on campus who are collecting data…can we share? Alumni, employer, student surveys Exit interviews with graduates Graduate follow-up studies Percentage of students who go onto graduate school Retention and transfer studies Job placement statistics Activities selected or elected by students Faculty/Student ratios Percentage of students who study abroad Enrollment trends Percentage of students who graduate within five-six years Diversity of student body

Do we want to collect direct or indirect evidence? students display their knowledge and skills students or someone else reflect student learning, behavior, attitudes rather than to demonstrate it

Regardless of data collection method make sure that the method… Is linked to values Measures outcomes that are tied to mission, goals, and objectives Takes into account any professional standards Measures expectations What students need to know, do, or report to assess your intended outcomes What the program will do to ensure what students will be able to do and to know. Process first, outcomes second. At a specified level, rather than just being satisfied

Regardless of data collection method make sure that the method… Answers important questions Collect data you believe will be useful to answering the important questions you have raised Organize reports around issues, not solely data Uses reliable and valid measurement Provides accurate and consistent information (e.g., primary vs secondary source, accuracy, credibility, and trustworthiness) measuring outcomes and appear credible to others Ensures compliance to the methods (e.g., will participants fill out questionnaires carefully, engage in interviews or focus groups) Provides enough diversity of information (triangulation, subgroups) to makes decisions about program and participants Respects human rights/ethics (confidentiality & anonymity, do no harm, appropriate use of data for intended purposes)

Regardless of data collection method make sure that the method… Creates data/information that informs Stakeholders and decision making Pedagogy, budgeting, planning, decision making, or policies. Considers who needs to make decisions with this data and the kinds of evidence for the decisions that need to be made Considers how information will be reported to convey main points and to inform Considers available resources Collected and analyzed in low-cost and practical manner (e.g., using questionnaires, surveys and checklists) Uses what time and resources are available Stays in budget and timeline Considers analysis and interpretation capabilities Fits into annual responsibilities