The Evolution of a QEP: One Institution’s Mid-Cycle Report Learning Without Borders: Internationalizing the Gator Nation University of Florida Gainesville, Florida Timothy S. Brophy, Professor and Director of Institutional Assessment Matthew Jacobs, Associate Professor and Director of Undergraduate Programs Cynthia Tarter, Assistant Director of Undergraduate Programs

Today’s learning outcomes Using UF’s QEP as a foundation, we will: Examine our process for modifying the QEP while it is being implemented; Discuss and critique the process of data analysis and plan modification; and Interpret the utility of the UF process model as it applies to your institution.

UF’s QEP: Learning Without Borders Institutional planning and development The initiatives Study Abroad On-campus course development Campus Life International Scholars Program International Calendar and Marketing

UF QEP: Student Learning Framework

Approach to implementation The “plan” and the “reality” The institutional level—integration and buy-in The personnel level—staffing assessments The faculty level—engagement and collaboration The student level—benefits and opportunities Director, QEP (Dean) Assistant Director (Admin) Graduate Assistant (Assessment) ? Director, Undergraduate Academic Programs Gold Silver

UF Process Model: Assessment and Data Reporting Assessment and Institutional Effectiveness Data Reporting Establish Mission, Goals, and Outcomes Assessment Planning Implement the Plan and Gather Data Interpret and Evaluate the Data Modify and Improve February – Assessment Plans submitted for the next AY October - Assessment Data, results, and use of results for previous AY reported

Process Model: UF Assessment System Develop Academic Assessment plans and data reports System entry: Submit for institutional review Implement plan and collect data System entry: Submit reports

Validity and Reliability

Validity Validity is “a unitary concept – it is the degree to which all the accumulated evidence supports the intended interpretation of test scores for the proposed use.” APA/AERA/NCME, Standards for Educational and Psychological Testing, 2014. For institutional assessment, the evidence is SLO data (the ‘test scores’) and the proposed use of this data is to determine the degree to which an SLO has been met by students in the program. Interpretation – the faculty set criteria for success, and make inferences from the SLO data as to the degree to which their students achieve the SLO.

Checking for Validity at the institutional level All plans and data reports are reviewed by Institutional Assessment staff All measures of goals and SLOs are reviewed to ensure that they lead to data pertinent to the goal or outcome (validity) If there are questions, the plan or report is returned for modification or clarification

Reliability/Precision In its broadest sense, “reliability refers to the consistency of scores across replications of a testing procedure…this is always important…and the need for precision increases as the consequences of decisions and interpretations grow in importance.” APA/AERA/NCME, Standards for Educational and Psychological Testing, 2014.

Checking for Reliability at the institutional level Reliability of SLO assessments is the responsibility of the academic program; in this case, the QEP staff Faculty have access to the built-in reliability functions of our Learning Management System (Canvas) – they can program the LMS to collect data on their program SLOs for content assessments We do monitor the reliability of our QEP measures, which are administered institutionally – institutionally developed measures (IntCrit and IntComm)

Getting to results: Data collection and analysis The process and challenges—incentives and obligations • Student surveys—By cohorts, program participants • Faculty—course SLO data • Identifying other sources of data Interpreting results • Were you asking the right questions? • Anomalies and outliers? • Factors inhibiting growth/change Key findings • Study abroad and curriculum enhancement grants • Applying global student learning outcomes across multiple disciplines • Student’s intercultural interaction and active learning of different cultural norms

Mid-cycle Modifications: Use of results More strategic approach Study abroad program development and course development Unit based vs. individual faculty; curriculum integration; diversity Marketing Different constituencies require different approaches On-campus engagement

Conclusions Busting myths OK to change and adapt based on assessment and analysis Faculty participation Campus-wide knowledge and engagement Energy intensive administration and outreach Anticipation and mitigation of challenges Flexibility, sensitivity to challenges Solution oriented and collaborative Use of results to modify and improve plan

Contact Information and Q&A Timothy S. Brophy, Director of Institutional Assessment and Professor, Music Education University of Florida Office of the Provost 352-273-4476 tbrophy@aa.ufl.edu Matthew F. Jacobs, Director of Undergraduate Academic Programs and the International Studies Program, Associate Professor of History University of Florida International Center and College of Liberal Arts and Sciences 352-273-4439 mjacobs@ufic.ufl.edu Cynthia M. Tarter, Assistant Director of Undergraduate Academic Programs and the International Studies Program