Academic and Administrative Assessment Information Session University of South Florida Office of Institutional Effectiveness and Review

Welcome and thank you! The Office of Institutional Effectiveness, Academic Planning and Review Melanie Wicinski, PhD Assistant Director Institutional Effectiveness & Assessment Programs Conduct Internal Assessments Plans/Reports submitted SAM Database Reviewed, Analyzed, and Compiled by OIE Required reports submitted to BOG, SACSCOC

Agenda Academic assessment reports/SAM academic report Components and requirements of an assessment plan Administrative assessment reports System for Assessment Management (SAM)

Academic Assessment Plans/Reports Annual report of each degree and certificate program. Program learning goals and learning outcomes Plans for future program development Formative assessment of the degree or certificate program, not summative assessment of students. Non-comprehensive Non-linear Program-focused Looking for opportunities for growth and development Continuous

Components of an Assessment Plan in SAM 1. Program/Department Mission Statement 2. Program Learning Goals (LGs) 3. Learning Outcomes (1 or more per LG) i. Statement of Learning Objective ii. Method of Assessment iii. Performance Targets iv. Results v. Use of Assessment Results

Components of an Assessment Plan in SAM Breakdown Mission Learning Goals Learning Outcome Learning Outcome Statement Method of Assessment Performance Targets Assessment Results Use of Assessment Results Components of an Assessment Plan in SAM Breakdown

Learning Goals v. Learning Outcomes A demonstrable skill, measureable change in behavior Specific Statements What students will know and demonstrate after the course(s) Example: “Majors in this program will be able to conduct original research in this discipline using appropriate methods.” Includes 5 detailed parts Not always measureable, broad statements Skills, Content, knowledge, or tasks students will have Example: “Development of Critical Thinking Skills.” Undergraduate Requirements Content Specific Knowledge Critical Thinking Skills Communication Skills *Graduate degrees and Certificates: no minimum reporting requirement

Method Of Assessment Section Should be the most detailed section under the Learning Outcome Less flexible, has several requirements of what should be included Examples of Assessment Methods Written Student Work Student Presentation/Performance Portfolio of Student Work Part of a Lab Report Embedded test questions Part of a thesis or dissertation Standardized tests* Internship/Practicum evaluation*

Method of Assessment: Requirements 1. A clear description of the assessment method 2. A statement on how the assessment specifically measures the task, information or competency stated in the Learning Outcome 3. Context of the Assessment 4. How the assessment will be scored 5. Which students in the program will be assessed

Method of Assessment: Requirements Cont. 6. If a sample of student work will be analyzed in lieu of all students, include information on the sample   7. If employing a rubric, information is needed on the rubric used and how it was developed and validated E.g. How Inter-rater reliability will be addressed 8. Information on how the rubric will be calibrated 9. Information on who will be reviewing and rating the assessment.

Method of Assessment: Inappropriate Assessments Grades CANNOT be used as outcomes (summative assessment of student work, not formative assessment of program work) Summative Student Assessments (the passing or failing of a culminating exam, course or program) CANNOT be used (summative vs. formative) Student/faculty/Alumni Surveys CANNOT be used (indirect measures) Publication in an academic journal CANNOT be used as a student learning outcome (external factors) Job placement CANNOT be used (external factors)

Performance Targets: Target for measured program performance SHORT statement on what benchmark(s) the program wants to meet This is not how we think our students will perform, but what we want our program to do Should be worded in terms of performance on the rubric or rating instrument “Performance target will be considered met if 75% of students achieve an overall score of 4/5 or higher” “Performance target will be considered met if 80% of students assessed will receive a final score of “Commendable” or higher.

Assessment Results SHORT statement on the final results of the assessment Stated in terms of the rubric or grading instrument Does not need to include commentary about the results and why or why not the performance targets were met Does not need to be, and should not be, perfect Don’t be afraid to report that the target was not met

Use of Assessment Results: Do Should include specified, actionable “next steps” the program will take, using the assessment results, to develop or improve on an programmatic level: Curriculum Mapping/redevelopment Revisit or revise the assessment method/rubric Revisions to plan of study/program structure Development of new modules/courses Faculty development

Use of Assessment Results: Don’t Remember: The responsibility is on the program, not its students. Use of results section should indicate programmatic improvements This is not an assessment report of student achievement Do not include: tutoring effort, sending students to the writing center, counseling students, providing students feedback, or improved recruitment

Certificates The following three possibilities apply to certificates: If ALL of the students in the certificate program are non-degree seeking, non-financial aid students, then you can state that in SAM and do not have to complete a program assessment. If even one student who is degree-seeking and receives financial aid is enrolled in the certificate program then you have to provide an assessment (I know some certificates can be taken as electives in a degree-seeking program and this is where this may come into play). If your certificate is a sub-set of a full degree program (the classes in the certificate are really just a portion of a full degree program), then your certificate assessment can be identical or a subset of your assessment reported for the degree program.

Rubric for Academic Program Assessment Cycle Element Quality Level Missing Data Initial Emerging Developed Highly Developed Planning Mission May be the departmental mission Mission statement is missing A statement is present, but is not a mission, but a description of the program Mission statement is present and appropriate Program Goals Should match the Mission Statement and each should be one complete statement Undergraduate Three must be covered (Content Specific Knowledge, Communication, Critical Thinking) Any one portion of the three required Goals are not completed (Content Specific Knowledge, Communication, Critical Thinking). A statement is present, but not a Goal Goals are succinctly and clearly stated Graduate and Certificate No required amount/content program goals No Goal present Not a learning Goal Learning Goals are listed, but not aligned with mission statement Learning Goals are listed and clearly aligned with the mission statement Learning Outcome Statement Should be actionable and realistic No Learning Outcome present or not revised from previous year The statement is present, but not stated as a learning outcome The Learning Outcome is present, but may not be an appropriate student learning outcome The Learning Outcome may need minor revisions The Learning Outcome is present and well-developed and clearly stated Method of Assessment Should include type of assessments, how the program is being measured, and should provide information specific to the stated measure No stated method of assessment or not revised from previous year Method is inappropriate; not connected or specific to the learning outcome; use of IDM, GO, external factors Method is appropriate, but may need further description, including more information about rubric, raters, or context Method may need minor clarification Method is clear, concise and well-defined. Meets all reporting requirements Performance Targets Should be thought of as PROGRAM benchmarks. Indicate what numbers would indicate that the program has met their programmatic goal No performance target present or targets do not correspond with methods Performance targets present, but stated in terms of assignment, course, or degree completion Perfomance targets are present, but may need minor clarification Performance targets are appropriate and well-stated Reporting Assessment Results Include specific findings, including number of students included in the assessment, and any other appropriate statistical information No Assessment Results present Assessment results present, but unclear how they relate to methods Assessment results present, but not specific Assessment results present and clear, but missing minor data (such as: number of students) Assessment results present and clear. All data reported appropriately Use of Assessment Results Should include actionable items for PROGRAM improvement No Use of Results present Use of Results statement indicates no changes are necessary Use of Results statement does not indicate what changes will be made based off data Use of Results statement is present and clear; may need minor clarification Use of Results statement is present and clear and specifically indicates the changes that are being made

The Phases of the Academic Reporting Cycle (Note: For all colleges, except Education and Business) Important Notes: Changes to your plan can be made at any time. Reviews by OIE are intended to be helpful – this is a conversation. Adapted from Rodriguez & Frederick, Miami Dade College 2013

Academic Assessment Reporting Academic Reporting Cycle Dates Note: For all colleges except Education and Business Academic Assessment Reporting Plan Year Plan Due Dates Plan Approval Date Final Report Date 2016 March 31, 2016 April 15, 2016 December 15, 2016 2017 January 15, 2017 January 30, 2017 December 15, 2017 2018 January 15, 2018 January 30, 2018 December 15, 2018

The Phases of the Academic Reporting Cycle (Note: For Education and Business ONLY) Important Notes: Changes to your plan can be made at any time. Reviews by OIE are intended to be helpful – this is a conversation. Adapted from Rodriguez & Frederick, Miami Dade College 2013

Academic Reporting Cycle Dates Note: For Education and Business ONLY Academic Assessment Reporting Plan Year Plan Due Dates Plan Approval Date Final Report Date 2015-2016 N/A October 1, 2016 2016-2017 October 15, 2016 October 31, 2016 October 1,2017 2017-2018 October 15, 2017 October 31, 2017 October 1,2018

Administrative Assessment Plans Very similar to Academic plans They are different in the following ways: Learning Outcomes vs. Objectives Same type of cycle, but different due dates Not nearly as strenuous and detailed – RIGHT NOW! May include graduation rates, alumni surveys, employer surveys, etc. Ask: What are the goals for your support area? What can you improve? What goals might you have been given from leadership?

Example of Administrative Assessment Objective Objective: Encourage funded and unfunded research involving community- based partnerships. Method: Track grant applications captured by Office [name] by using [software]. Performance Target: Increase community-based grant applications by 5%. Results: There were ### community-based grant proposals submitted this past calendar year, representing a #.#% increase in proposals.  There were ### community-based grant awards in that year, representing an #.#% in awards, to a total of $#.  Already in 2016, there is a total of $# in awards. Use of Results: These data are also self-identified on the … Form of the Office [name].  A task force is working on easier methods of gathering and recording community-based grants and funding, as well as community-based research that is done without external funding.

The Phases of the Administrative Reporting Cycle Important Notes: Changes to your plan can be made at any time. Reviews by OIE are intended to be helpful – this is a conversation. Adapted from Rodriguez & Frederick, Miami Dade College 2013

Administrative Reporting Cycle Dates

“SAM” The System for Assessment Management How does USF organize academic and administrative program assessment plans? “SAM” The System for Assessment Management Database maintained by the Office of Institutional Effectiveness Available online with NetID/USF password

Accessing the System for Assessment Management The pink (or orange) paper provided in your training materials has detailed instructions for logging in. Recommendation: Once you get to the login page, bookmark it!

How do you know what your plan was rated?

Overall Plan Rating Scale If a plan receives a yellow or red overall rate, we will select that is “not complete” Programs will be notified and asked to make changes to sections that still need clarification or modification When secondary adjustments have been made, the “Plan complete?” button should be changed to “Yes”

Take Away Points Formative of program, not summative of students Not comprehensive assessment of all program efforts, make it manageable Use samples when possible Not linear, if one area has met the performance target and is going well, move on to another area Develop a tentative five year assessment progression Brainstorm possible area of improvement to tackle over the next 5 years University-wide 5 year strategic plans will be coming to the college and unit level.

PLEASE CONTACT OUR OFFICE ALONG THE WAY! We are here to help! Questions? PLEASE CONTACT OUR OFFICE ALONG THE WAY! We are here to help!