1. Faculty Senate Assessment Committee Facilitator: Katherine Cermak Associate Dean for Planning & Assessment April 2016

2. Today’s Outcomes Participants will be able to: Use basic techniques to summarize assessment data. Quantitative Qualitative Compare assessment data to expectations of student knowledge or ability. Present assessment data as information to support decision-making.

3. Assessment Cycle 2) Locate demonstrations of outcome(s) and collect student work products. 3) Analyze student work and determine to what extent students are meeting expectations. 4) Share and discuss results internally. 5) Determine (if appropriate) actions for program improvement (and also the assessment activities) 1) Select learning outcome(s) to be assessed. Effectively summarize and present results

4. Basic ways to summarize Quantitative Tallies/Counts Mean Average Percentages Qualitative Thematic Coding

5. Count/Tally Outcome: Students will develop, organize, and communicate information. Poor work— not acceptable Developing-- Approaching Expectations Proficient-- Meeting Expectations Exceeding Expectations Appropriate use of sources 232520 Integrated review of the literature 282812 Well-reasoned choice of methodologies 28337 Appropriate analysis 5102510 Correct interpretation of results 10 255 n=50

6. Count/Tally Outcome: develop, organize, and communicate information within the discipline Poor work—not acceptable Developing-- Approaching Expectations Proficient -- Meeting Expectations Exceeding Expectations Meeting or Exceeding Expectations Appropriate use of sources 23252045/50 Integrated review of the literature 28281240/50 Well-reasoned choice of methodologies 2833740/50 Appropriate Analysis 510251035/50 Correct interpretation of results 10 25530/50

7. Average Outcome: develop, organize, and communicate information within the discipline AverageStandard Deviation Appropriate use of sources 3.30.75 Integrated review of the literature 30.76 Well-reasoned choice of methodologies 2.90.68 Appropriate Analysis 2.80.88 Correct interpretation of results 2.50.93 Overall Outcome 2.8.99 Poor=1/Developing=2/Proficient=3/Exceeding=4

8. Percentages Outcome: Communicate information Poor work—not acceptable Developing -- Approaching Expectations Proficient -- Meeting Expectations Exceeding Expectations Meeting or Exceeding Expectations Appropriate use of sources 4%6%50%40%90% Integrated review of the literature 4%16%56%24%80% Well-reasoned choice of methodologies 4%16%66%14%80% Appropriate Analysis 10%20%50%20%70% Correct interpretation of results 20% 50%10%60% n=50 The expectation was that 80% of students would meet expectations and less than 5% of students would be in the poor category....

9. Multiple Choice Count/Tally Students can solve problems using scientific processes. Counts and Bar Chart

10. Multiple Choice Question Student can solve problems using scientific reasoning Percentages Expectation: 65% would select the correct answer Overall 60% chose the correct answer. Per item 56% - 67%. Performance expectations were not met. Question 16 To choose C students must.... A was the most common distractor because.... B was most likely chosen because....

11. Qualitative Data

12. Qualitative Data--Quick Quick/Exploratory Analysis Identify your preconceptions Quickly read through Random subsample if necessary Work with 1 or more colleagues Describe prominent suggestions, main points, common themes.

13. Qualitative Data--Quick Reporting Quick/Exploratoy Analysis Describe your process #of documents/comments/participants Analysis Method Findings Useful Phrases: The main issues discussed/mentioned were..., The prevailing factor/theme was... XYZ was a common theme raised... Less useful phrases (but sometimes necessary): A small number... One respondent...

14. Qualitative Data—In Depth Coding Identify your expectations/biases Read through all documents Read through again Identify themes Code —by discrete inputs (each comment/each line). Examine your “other” or “misc” category for additional themes. Examine text that has no code for missing themes Tabulate Prevalence (does not equal importance) Interpret

17. Qualitative Data—In Depth Code TallyInterpretationQuotes Rubrics—positive (setting expectations) 5The sharing of the rubric clarified expectations; especially to students, but also among faculty members. “…” Rubrics—positive (grading) 4Faster, more efficient grading.“…” Rubrics—positive (consensus building) 3Creating and norming together created more consensus around expectations/grading “…” Rubrics—positive (planning/self- assessment) 3Some students used the rubric to improve their performance. “…” Rubrics—positive (rigor maintained) 2Sharing the rubrics didn’t result in the assignments becoming less rigorous. “…” Rubrics—negative (lack of consensus) 2Lack of consensus on the criteria themselves and relative importance—divisive. “…”

18. Resources Qualitative Data Analysis Software Descriptions/Cost http://www.eval.org/p/cm/ld/fid=81 Google Forms/Spreadsheets (Teaching Technology) Assistance Dr. Julie Zhu, Deputy Director for Instructional Design and Technology Integration Suskie, Linda. (2009). Assessing Student Learning: A Common Sense Guide. San Francisco, CA: Jossey-Bass. Using Assessment Results https://manoa.hawaii.edu/assessment/workshops/index.htm

19. Contacts Faculty Senate Assessment Committee Members Engineering & Weapons: Dr. Steve Graham and Dr. Deborah Mechtel Humanities & Social Sciences: Dr. Michelle Allen-Emerson and Dr. Silvia Peart Math & Science: Dr. Nick Frigo and Dr. Shirley Lin Professional Development: LT C. Hirsch, LT C. Roncketti (incoming) Leadership Education & Development: CDR Joe McInerney, CDR Lon Olson (incoming) Office of the Academic Dean & Provost Dr. Katherine Cermak Website: www.usna.edu/Academics/Academic-Dean/Assessment/www.usna.edu/Academics/Academic-Dean/Assessment/ Assessment Resources One-on-One consultations with departments, faculty, and staff Yard-wide assessment events