1. Academic and Administrative Assessment Information Session
University of South Florida
Office of Institutional Effectiveness and Review

2. Welcome and thank you!
The Office of Institutional Effectiveness, Academic Planning and Review
Melanie Wicinski, PhD
Assistant Director
Institutional Effectiveness & Assessment
Programs Conduct Internal Assessments
Plans/Reports submitted
SAM Database
Reviewed, Analyzed, and Compiled by OIE
Required reports submitted to
BOG, SACSCOC

3. Agenda Academic assessment reports/SAM academic report
Components and requirements of an assessment plan
Administrative assessment reports
System for Assessment Management (SAM)

4. Academic Assessment Plans/Reports
Annual report of each degree and certificate program.
Program learning goals and learning outcomes
Plans for future program development
Formative assessment of the degree or certificate program, not summative assessment of students.
Non-comprehensive
Non-linear
Program-focused
Looking for opportunities for growth and development
Continuous

5. Components of an Assessment Plan in SAM
1. Program/Department Mission Statement
2. Program Learning Goals (LGs)
3. Learning Outcomes (1 or more per LG)
i. Statement of Learning Objective
ii. Method of Assessment
iii. Performance Targets
iv. Results
v. Use of Assessment Results

6. Components of an Assessment Plan in SAM Breakdown
Mission
Learning Goals
Learning Outcome
Learning Outcome Statement
Method of Assessment
Performance Targets
Assessment Results
Use of Assessment Results
Components of an Assessment Plan in SAM
Breakdown

7. Learning Goals v. Learning Outcomes
A demonstrable skill, measureable change in behavior
Specific Statements
What students will know and demonstrate after the course(s)
Example: “Majors in this program will be able to conduct original research in this discipline using appropriate methods.”
Includes 5 detailed parts
Not always measureable, broad statements
Skills, Content, knowledge, or tasks students will have
Example: “Development of Critical Thinking Skills.”
Undergraduate Requirements
Content Specific Knowledge
Critical Thinking Skills
Communication Skills
*Graduate degrees and Certificates:
no minimum reporting requirement

8. Method Of Assessment Section
Should be the most detailed section under the Learning Outcome
Less flexible, has several requirements of what should be included
Examples of Assessment Methods
Written Student Work
Student Presentation/Performance
Portfolio of Student Work
Part of a Lab Report
Embedded test questions
Part of a thesis or dissertation
Standardized tests*
Internship/Practicum evaluation*

9. Method of Assessment: Requirements
1. A clear description of the assessment method
2. A statement on how the assessment specifically measures the task, information or competency stated in the Learning Outcome
3. Context of the Assessment
4. How the assessment will be scored
5. Which students in the program will be assessed

10. Method of Assessment: Requirements Cont.
6. If a sample of student work will be analyzed in lieu of all students, include information on the sample  
7. If employing a rubric, information is needed on the rubric used and how it was developed and validated
E.g. How Inter-rater reliability will be addressed
8. Information on how the rubric will be calibrated
9. Information on who will be reviewing and rating the assessment.

11. Method of Assessment: Inappropriate Assessments
Grades CANNOT be used as outcomes (summative assessment of student work, not formative assessment of program work)
Summative Student Assessments (the passing or failing of a culminating exam, course or program) CANNOT be used (summative vs. formative)
Student/faculty/Alumni Surveys CANNOT be used (indirect measures)
Publication in an academic journal CANNOT be used as a student learning outcome (external factors)
Job placement CANNOT be used (external factors)

12. Performance Targets: Target for measured program performance
SHORT statement on what benchmark(s) the program wants to meet
This is not how we think our students will perform, but what we want our program to do
Should be worded in terms of performance on the rubric or rating instrument
“Performance target will be considered met if 75% of students achieve an overall score of 4/5 or higher”
“Performance target will be considered met if 80% of students assessed will receive a final score of “Commendable” or higher.

13. Assessment Results
SHORT statement on the final results of the assessment
Stated in terms of the rubric or grading instrument
Does not need to include commentary about the results and why or why not the performance targets were met
Does not need to be, and should not be, perfect
Don’t be afraid to report that the target was not met

14. Use of Assessment Results: Do
Should include specified, actionable “next steps” the program will take, using the assessment results, to develop or improve on an programmatic level:
Curriculum Mapping/redevelopment
Revisit or revise the assessment method/rubric
Revisions to plan of study/program structure
Development of new modules/courses
Faculty development

15. Use of Assessment Results: Don’t
Remember: The responsibility is on the program, not its students. Use of results section should indicate programmatic improvements
This is not an assessment report of student achievement
Do not include: tutoring effort, sending students to the writing center, counseling students, providing students feedback, or improved recruitment

16. Certificates The following three possibilities apply to certificates:
If ALL of the students in the certificate program are non-degree seeking, non-financial aid students, then you can state that in SAM and do not have to complete a program assessment.
If even one student who is degree-seeking and receives financial aid is enrolled in the certificate program then you have to provide an assessment (I know some certificates can be taken as electives in a degree-seeking program and this is where this may come into play).
If your certificate is a sub-set of a full degree program (the classes in the certificate are really just a portion of a full degree program), then your certificate assessment can be identical or a subset of your assessment reported for the degree program.

17. Rubric for Academic Program Assessment
Cycle
Element
Quality Level
Missing Data
Initial
Emerging
Developed
Highly Developed
Planning
Mission May be the departmental mission
Mission statement is missing
A statement is present, but is not a mission, but a description of the program
Mission statement is present and appropriate
Program Goals Should match the Mission Statement and each should be one complete statement
Undergraduate Three must be covered (Content Specific Knowledge, Communication, Critical Thinking)
Any one portion of the three required Goals are not completed (Content Specific Knowledge, Communication, Critical Thinking).
A statement is present, but not a Goal
Goals are succinctly and clearly stated
Graduate and Certificate No required amount/content program goals
No Goal present
Not a learning Goal
Learning Goals are listed, but not aligned with mission statement
Learning Goals are listed and clearly aligned with the mission statement
Learning Outcome Statement Should be actionable and realistic
No Learning Outcome present or not revised from previous year
The statement is present, but not stated as a learning outcome
The Learning Outcome is present, but may not be an appropriate student learning outcome
The Learning Outcome may need minor revisions
The Learning Outcome is present and well-developed and clearly stated
Method of Assessment Should include type of assessments, how the program is being measured, and should provide information specific to the stated measure
No stated method of assessment or not revised from previous year
Method is inappropriate; not connected or specific to the learning outcome; use of IDM, GO, external factors
Method is appropriate, but may need further description, including more information about rubric, raters, or context
Method may need minor clarification
Method is clear, concise and well-defined. Meets all reporting requirements
Performance Targets Should be thought of as PROGRAM benchmarks. Indicate what numbers would indicate that the program has met their programmatic goal
No performance target present or targets do not correspond with methods
Performance targets present, but stated in terms of assignment, course, or degree completion
Perfomance targets are present, but may need minor clarification
Performance targets are appropriate and well-stated
Reporting
Assessment Results Include specific findings, including number of students included in the assessment, and any other appropriate statistical information
No Assessment Results present
Assessment results present, but unclear how they relate to methods
Assessment results present, but not specific
Assessment results present and clear, but missing minor data (such as: number of students)
Assessment results present and clear. All data reported appropriately
Use of Assessment Results Should include actionable items for PROGRAM improvement
No Use of Results present
Use of Results statement indicates no changes are necessary
Use of Results statement does not indicate what changes will be made based off data
Use of Results statement is present and clear; may need minor clarification
Use of Results statement is present and clear and specifically indicates the changes that are being made

18. The Phases of the Academic Reporting Cycle (Note: For all colleges, except Education and Business)
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Adapted from Rodriguez & Frederick, Miami Dade College 2013

19. Academic Assessment Reporting
Academic Reporting Cycle Dates Note: For all colleges except Education and Business
Academic Assessment Reporting
Plan Year
Plan Due Dates
Plan Approval Date
Final Report Date
2016
March 31, 2016
April 15, 2016
December 15, 2016
2017
January 15, 2017
January 30, 2017
December 15, 2017
2018
January 15, 2018
January 30, 2018
December 15, 2018

20. The Phases of the Academic Reporting Cycle (Note: For Education and Business ONLY)
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Adapted from Rodriguez & Frederick, Miami Dade College 2013

21. Academic Reporting Cycle Dates Note: For Education and Business ONLY
Academic Assessment Reporting
Plan Year
Plan Due Dates
Plan Approval Date
Final Report Date
N/A
October 1, 2016
October 15, 2016
October 31, 2016
October 1,2017
October 15, 2017
October 31, 2017
October 1,2018

22. Administrative Assessment Plans
Very similar to Academic plans
They are different in the following ways:
Learning Outcomes vs. Objectives
Same type of cycle, but different due dates
Not nearly as strenuous and detailed – RIGHT NOW!
May include graduation rates, alumni surveys, employer surveys, etc.
Ask: What are the goals for your support area? What can you improve? What goals might you have been given from leadership?

23. Example of Administrative Assessment Objective
Objective: Encourage funded and unfunded research involving community- based partnerships.
Method: Track grant applications captured by Office [name] by using [software].
Performance Target: Increase community-based grant applications by 5%.
Results: There were ### community-based grant proposals submitted this past calendar year, representing a #.#% increase in proposals.  There were ### community-based grant awards in that year, representing an #.#% in awards, to a total of $#.  Already in 2016, there is a total of $# in awards.
Use of Results: These data are also self-identified on the … Form of the Office [name].  A task force is working on easier methods of gathering and recording community-based grants and funding, as well as community-based research that is done without external funding.

25. The Phases of the Administrative Reporting Cycle
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Adapted from Rodriguez & Frederick, Miami Dade College 2013

26. Administrative Reporting Cycle Dates

27. “SAM” The System for Assessment Management
How does USF organize academic and administrative program assessment plans?
“SAM” The System for Assessment Management
Database maintained by the Office of Institutional Effectiveness
Available online with NetID/USF password

28. Accessing the System for Assessment Management
The pink (or orange) paper provided in your training materials has detailed instructions for logging in.
Recommendation: Once you get to the login page, bookmark it!

30. How do you know what your plan was rated?

31. Overall Plan Rating Scale
If a plan receives a yellow or red overall rate, we will select that is “not complete”
Programs will be notified and asked to make changes to sections that still need clarification or modification
When secondary adjustments have been made, the “Plan complete?” button should be changed to “Yes”

32. Take Away Points Formative of program, not summative of students
Not comprehensive assessment of all program efforts, make it manageable
Use samples when possible
Not linear, if one area has met the performance target and is going well, move on to another area
Develop a tentative five year assessment progression
Brainstorm possible area of improvement to tackle over the next 5 years
University-wide 5 year strategic plans will be coming to the college and unit level.

33. PLEASE CONTACT OUR OFFICE ALONG THE WAY! We are here to help!
Questions?
PLEASE CONTACT OUR OFFICE ALONG THE WAY!
We are here to help!