1. How to Effectively and Efficiently Conduct and Use Annual Assessment
Director, Office of Academic Program Assessment (OAPA)
Dr. Amy Liu, Professor of Sociology
Assessment Consultants, OAPA
Dr. Jacqueline Brooks, Professor of Sociology
Dr. Chia-Jung Chung, Professor of Education
Dr. Milica Markovic, Professor of Engineering
California State University, Sacramento
Spring 2019

2. Outline The Annual Assessment Process, OAPA, and Your Program
Feedback Report and Appendices as Resources
How to Use the Feedback Reports to Improve Learning & Success
Assessment Template: Simple, Clear, and Useful
Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions

3. Office of Academic Program Assessment (OAPA)
Run by faculty to provide guidance and feedback on program assessment
Help Academic Affairs
Program Review
WASC
Academic Strategic Planning & Resource Allocation
Help Programs
- To make Annual Assessment: simple, clear, and useful (high quality)
- To connect to the university strategic plan and assess the program strategic plan

4. How OAPA uses annual assessment?
Provide feedback to programs/departments to improve:
student learning and success
annual assessment
program review
strategic planning and resource allocation
Successfully complete University and Department accreditations
WASC, ABET, AACSB, or CCTC
Develop professional development workshops and FLCs
Assess strategic plan and use action plan for improvement

5. Outline The Annual Assessment Process, OAPA, and Your Program
Feedback Report and Appendices as Resources
How to Use the Feedback Reports to Improve Learning & Success
Assessment Template: Simple, Clear, and Useful
Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions

6. Assessment Principles Assessment—an ongoing, interactive process using backward design and five basic principles:
Define Learning Goals
Outcomes & Definitions
Curriculum Map
Design Instruction
Help Students Achieve Outcomes
Decide on Assessments
Evidence of Learning Outcomes
“Backward Design” for Assessment (Outcomes & Definitions)
Plan step 1
PLOs:
-Select
-Define
-Curriculum map
Plan step 2
Expectations 1:
-Develop or adopt Rubrics
_ Define Standards of performance
Plan step 3
Methods and Measures
-Assignments
-Tests
-Projects
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop
Q 2.2

7. 2. Graduate Learning Goals (GLGs) (Appendix 9 in 2017-2018 Feedback)
Appendices 8-9 Baccalaureate Learning Goals, Graduate Learning Goals, and AAC&U VALUE Rubrics (Simple)
Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics.
1. Baccalaureate Learning Goals (BLGs) (Appendix 8 in Feedback)
2. Graduate Learning Goals (GLGs) (Appendix 9 in Feedback)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
1.3 Intellectual and Practical Skills
2.3 Intellectual and Practical Skills
3.3 Intellectual and Practical Skills
1.4 Personal and Social Responsibility (Values)
2.4 Personal and Social Responsibility (Values)
3.4 Personal and Social Responsibility (Values)
1.5 Integrative Learning
3.5 Integrative Learning
2.6 Research 7

8. Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Detailed)
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees)
5. Bloom’s Revised Taxonomy
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
4.1 Specialized Knowledge
5.1 Bloom’s Revised Taxonomy
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
4.2 Broad and Integrative Knowledge
5.2 Bloom’s Revised Taxonomy
1.3 Intellectual and Practical Skills
1.3.1 Communication
1.3.1a Written
1.3.1b Oral
1.3.2 Critical Thinking
1.3.3 Literacy
1.3.3a Information
1.3.3b Quantitative
1.3.4 Inquiry and Analysis
1.3.5 Creative Thinking
1.3.6 Reading
1.3.7 Teamwork
1.3.8 Problem Solving
2.3 Intellectual and Practical Skills
2.3.1 Communication
2.3.2 Critical thinking/analysis
2.3.3 Information literacy
3.3 Intellectual and Practical Skills
3.3.1 Communication
3.3.1a Written
3.3.1b Oral
3.3.2 Critical Thinking
3.3.3 Literacy
3.3.3a Information
3.3.3b Quantitative
3.3.4 Inquiry and Analysis
3.3.5 Creative Thinking
3.3.6 Reading
3.3.7 Teamwork
3.3.8 Problem Solving
4.3 Intellectual Skills
5.3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education)
1.4 Personal and Social Responsibility (Values)
1.4.1 Global Learning
1.4.2 Intercultural Knowledge and Competence
1.4.3 Civic Knowledge and Engagement
1.4.4 Ethical Reasoning
1.4.5 Foundations and Skills for Lifelong Learning
2.4 Personal and Social Responsibility (Values)
2.4.1 Intercultural/global perspectives
2.4.6 Professionalism
3.4 Personal and Social Responsibility (Values)
3.4.1 Global Learning
3.4.2 Intercultural Knowledge and Competence
3.4.3 Civic Knowledge and Engagement
3.4.4 Ethical Reasoning
3.4.5 Foundations and Skills for Lifelong Learning
5.4 AAC&U VALUE Rubrics
1.5 Integrative Learning
3.5 Integrative Learning
4.5 Applied and Collaborative Learning
5.5 AAC&U VALUE Rubrics
2.6 Research
4.6 Civic and Global Learning
5.6 Civic and Global Learning 8

9. Initial Emerging Developed Highly Developed
Appendix 3: WSCUC “Rubric for Assessing the Quality of Academic Program Learning Outcomes” 9
Criterion
Initial
Emerging
Developed
Highly Developed
1. Comprehensive List
The list of outcomes is problematic: e.g., very incomplete, overly detailed, inappropriate, and disorganized. It may include only discipline-specific learning, ignoring relevant institution-wide learning. The list may confuse learning processes (e.g., doing an internship) with learning outcomes (e.g., application of theory to real-world problems).
The list includes reasonable outcomes but does not specify expectations for the program as a whole. Relevant institution-wide learning outcomes and/or national disciplinary standards may be ignored. Distinctions between expectations for undergraduate and graduate programs may be unclear.
The list is a well-organized set of reasonable outcomes that focus on the key knowledge, skills, and values students learn in the program. It includes relevant institution-wide outcomes (e.g., communication or critical thinking skills). Outcomes are appropriate for the level (undergraduate vs. graduate); national disciplinary standards have been considered.
The list is reasonable, appropriate, and comprehensive, with clear distinctions between undergraduate and graduate expectations, if applicable. National disciplinary standards have been considered. Faculty has agreed on explicit criteria for assessing students’ level of mastery of each outcome.
2. Assessable Outcomes
Outcomes statements do not identify what students can do to demonstrate learning. “Statements understand scientific method” do not specify how understanding can be demonstrated and assessed.
Most of the outcomes indicate how students can demonstrate their learning.
Each outcome describes how students can demonstrate learning, e.g., “Graduates can write reports in APA style” or “Graduate can make original contributions to biological knowledge.”
Outcomes describe how students can demonstrate their learning. Faculty has agreed on explicit criteria statements such as rubrics, and have identified examples of student performance at varying levels of each outcome.
3. Alignment
There is no clear relationship between the outcomes and the curriculum that students experience.
Students appear to be given reasonable opportunities to develop the outcomes in the required curriculum.
The curriculum is designed to provide opportunities for students to learn and to develop increasing sophistication with respect to each outcome. This design may be summarized in a curriculum map.
Pedagogy, grading, the curriculum, relevant student support services, and co- curriculum are explicitly and intentionally aligned with each outcome. Curriculum map indicates increasing levels of proficiency.
4. Assessment Planning
There is no formal plan for assessing each outcome.
The program relies on short-term planning, such as selecting which outcome(s) to assess in current year.
The program has a reasonable, multi-year assessment plan that identifies when each outcome will be assessed. The plan may explicitly include analysis and implementation of improvements.
The program has a fully-articulated, sustainable, multi-year assessment plan that describes when and how each outcome will be assessed and how improvements based on findings will be implemented. The plan is routinely examined and revised, as needed.
5. The Student Experience
Students know little or nothing about the overall outcomes of the program. Communication of outcomes to students, e.g., in syllabi or catalog, is spotty or nonexistent.
Students have some knowledge of program outcomes. Communication is occasional and informal, left to individual faculty or advisors.
Students have a good grasp of program outcomes. They may use them to guide their own learning. Outcomes are included in most syllabi and are readily available in the catalog, on the web page, and elsewhere.
Students are well-acquainted with program outcomes and may participate in creation and use of rubrics. They are skilled at self-assessing in relation to the outcome levels of performance. Program policy calls for inclusion of outcomes in all course syllabi, and they are readily available in other program documents.

10. Outline The Annual Assessment Process, OAPA, and Your Program
Feedback Report and Appendices as Resources
How to Use the Feedback Reports to Improve Learning & Success
Assessment Template: Simple, Clear, and Useful
Report One Outcome in Detail: How to Answer the Open-Ended Assessment Questions

11. Make 2018-2019 Assessment Simple, Clear, and Useful
Section 1 – Report all outcomes assessed in
Q1.1, Q1.3a
Section 2 – Report one learning outcome in detail
Q 2.1-Q5.3.1
Section 3 – Report other assessment activities
Q6-Q9.1
Section 4 – Background information about the program
Q10-Q24 11

12. What do you want your students to know, to do, and to value?
Section 1 – Report all outcomes assessed in
What do you want your students to know, to do, and to value?

14. Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Detailed)
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees)
5. Bloom’s Revised Taxonomy
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
4.1 Specialized Knowledge
5.1 Bloom’s Revised Taxonomy
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
4.2 Broad and Integrative Knowledge
5.2 Bloom’s Revised Taxonomy
1.3 Intellectual and Practical Skills
1.3.1 Communication
1.3.1a Written
1.3.1b Oral
1.3.2 Critical Thinking
1.3.3 Literacy
1.3.3a Information
1.3.3b Quantitative
1.3.4 Inquiry and Analysis
1.3.5 Creative Thinking
1.3.6 Reading
1.3.7 Teamwork
1.3.8 Problem Solving
2.3 Intellectual and Practical Skills
2.3.1 Communication
2.3.2 Critical thinking/analysis
2.3.3 Information literacy
3.3 Intellectual and Practical Skills
3.3.1 Communication
3.3.1a Written
3.3.1b Oral
3.3.2 Critical Thinking
3.3.3 Literacy
3.3.3a Information
3.3.3b Quantitative
3.3.4 Inquiry and Analysis
3.3.5 Creative Thinking
3.3.6 Reading
3.3.7 Teamwork
3.3.8 Problem Solving
4.3 Intellectual Skills
5.3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education)
1.4 Personal and Social Responsibility (Values)
1.4.1 Global Learning
1.4.2 Intercultural Knowledge and Competence
1.4.3 Civic Knowledge and Engagement
1.4.4 Ethical Reasoning
1.4.5 Foundations and Skills for Lifelong Learning
2.4 Personal and Social Responsibility (Values)
2.4.1 Intercultural/global perspectives
2.4.6 Professionalism
3.4 Personal and Social Responsibility (Values)
3.4.1 Global Learning
3.4.2 Intercultural Knowledge and Competence
3.4.3 Civic Knowledge and Engagement
3.4.4 Ethical Reasoning
3.4.5 Foundations and Skills for Lifelong Learning
5.4 AAC&U VALUE Rubrics
1.5 Integrative Learning
3.5 Integrative Learning
4.5 Applied and Collaborative Learning
5.5 AAC&U VALUE Rubrics
2.6 Research
4.6 Civic and Global Learning
5.6 Civic and Global Learning

15. Think-Pair-Share
Circle all the program learning outcomes assessed in your program.
Where does each PLO belong in the BLG and GLG?
Where does each PLO belong in the detailed curriculum map?

16. Think-Pair-Share
How is that outcome practiced in your detailed curriculum map?
Are your students struggling with the thesis or critical thinking? If yes, what do you plan to do?

17. Other than GPA, What Data Are Used to Measure PLOs
Table 5: Sociology Evidence Map at the Program Level:
Other than GPA, What Data Are Used to Measure PLOs
Outcomes
Year
Outcome 1:
Advanced Sociological knowledge
Outcome 2:
Sociological contributions and applications
Outcome 3:
Challenges in sociology
Outcome 4:
Oral
Communication
Outcome 5:
Written
Outcome 6:
Critical
Thinking
Outcome 7:
Intercultural Knowledge & Competence
Outcome 8: Integration: Inquiry and Analysis
2012 – 2013
(program review)
Thesis
Research paper
Oral exam (thesis hearing)?
Exit survey?
Thesis defense
Self-study

18. An Example: The Curriculum Map for the Sociology Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”.
Outcomes
Classes
Outcome 1:
Advanced Sociological Knowledge
Outcome 2:
Sociological Contributions and Applications
Outcome 3:
Challenges in Sociology
Outcome 4:
Oral
Communication
Outcome 5:
Written
Outcome 6:
Critical
Thinking
Outcome 7:
Intercultural Knowledge & Competence
Outcome 8: Integration: Inquiry and Analysis
Required Classes
SOC 200A (Introduction)
I 1 I
SOC 200B (Thesis Prep.)
D,M
SOC 214 (Methods) D
I, D
SOC 215 (Analysis)
D, M
SOC 235 (Soc. Psych.)
SOC 240 (Theory)
SOC 500/599 (Thesis Work) M
Elective Classes
SOC 210 (Urban Sociology)
 D,M
SOC 220 (Change)
SOC 225 (Stratification)
SOC 226 (Gender)
SOC 230 (Social Org.)  D
SOC 238 (Environmental)
SOC 225 (Crime)
SOC 260 (Contem. Issues.)
SOC 265 (Race)
SOC 266 (Family)

19. The Detailed Curriculum/Evidence Map for the Sociology Graduate Program:
Aligning (Linking) Graduate Program Learning Outcomes to Key Assignments in Each Course in the Curriculum
Outcomes
Classes
Outcome 1:
Advanced Sociological knowledge
Outcome 2:
Sociological contributions and applications
Outcome 3:
Challenges in sociology
Outcome 4:
Oral
Communication
Outcome 5:
Written
Outcome 6:
Critical
Thinking
Outcome 7:
Intercultural Knowledge & Competence
Outcome 8: Integration: Inquiry and Analysis
Required Classes
SOC 200A (Introduction)
Reflection papers
Discussion and presentation
SOC 200B (Thesis Prep.)
Thesis proposal
1. Thesis proposal
2. Oral presentation
2. Q5 in student survey
2. Q6 in student survey
2. Q7 in student survey
SOC 214 (Methods)
Research proposal
SOC 215 (Analysis)
Data analysis
Class presentation
Data analysis assignments
2 data analysis papers
SOC 235 (Soc. Psych.)
Annotation/literature review
Annotation
Literature review
SOC 240 (Theory)
Research paper
3 papers/exam
Weekly paper
Global perspectives
20 page papers
SOC 500/599 (Theses)
Thesis
Elective Classes
SOC 210 (Urban Soc.)
Projects
2-3 class presentations
4 papers
SOC 220 (Change)
SOC 225 (Stratification)
2 essay exams
Essay exams
SOC 226 (Gender)
Research paper/weekly paper
Short weekly paper
SOC 230 (Social Org.)
 Research papers
Research papers
SOC 238 (Environmental)
Group/class projects
20 page paper
SOC 225 (Crime)
Class projects
Group/class project
Cultural comparison
SOC 260 (Contem. Issues. In Middle East)
Papers
SOC 265 (Race)
Research paper/
6-8 critical review
Critical review
Content
SOC 266 (Family)
Weekly short paper

20. An Example: The Curriculum Map for a Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”.
Outcomes
Classes
Outcome 1
Outcome 2
Outcome 3
Outcome 4
Outcome 5
Outcome 6
Outcome 7
Outcome 8
Required Classes
CLASS 1
I,D D
CLASS 2
CLASS 3
D,M M
CLASS 4
CLASS 5 I
CLASS 6
CLASS 7
CLASS 8
CLASS 9
CLASS 10
CLASS 11
CLASS 12
CLASS 13
CLASS 14
CLASS 15
CLASS 16
CLASS 17
CLASS 18 20

21. Section 2 – Report one learning outcome in detail

22. Outline for one PLO: The PLO and Its Definition (Q2.1 and Q2.1.1)
The Rubric(s) and Standard(s) of Performance/Expectations (Q2.2a)
The Direct Measure(s) (Q3.3.2)
The Data Table(s) (Q4.1)
The Assessment Plan = Action Plan (Q5.1.1-Q5.1.2) 22

23. PLO 6: Critical Thinking in the Sociology Graduate Program
6: Sociology graduate students will demonstrate a habit of systematically exploring social issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion” (Learning Goal/Outcomes): they will (PLO 6: Critical thinking adopted from the VALUE rubric):
6.1: Clearly identify and state the social issue/problem that needs to be considered critically, comprehensively describe the social issue/problem, and deliver all relevant information so it is necessary for a full understanding of the issue/problem (Explanation of issues);
6.2: Thoroughly interpret and evaluate sociological perspectives, theories, social methods/statistics, and any other current, credible, and relevant information (research, knowledge, tools, and/or views) to complete a thorough review of relevant literature and the problems to develop a comprehensive analysis or synthesis (Evidence);
6.3: Thoroughly analyze their own and others' assumptions and carefully evaluate the relevance of contexts when presenting a position (Influence of context and assumptions);
6.4: Consider the complexities (all sides) of a social issue. Limits of position and others' points of view are acknowledged and synthesized within position (Student's position);
6.5: Form conclusions, consequences and implications that are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order (Conclusions and related outcomes). 23

24. Outline for one PLO: The PLO and Its Definition (Q2.1 and Q2.1.1)
The Rubric(s) and Standard(s) of Performance/Expectations (Q2.2a)
The Direct Measure(s) (Q3.3.2)
The Data Table(s) (Q4.1)
The Assessment Plan = Action Plan (Q5.1.1-Q5.1.2) 24

25. The VALUE Rubric for the Critical Thinking Skill
Criterion
Exceed Standards 4
Meet Standards 3
Meet Minimum Standards 2
Don’t Meet Standards 1
6.1: Explanation of issues
Issue/problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding.
Issue/problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions.
Issue/problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown.
Issue/problem to be considered critically is stated without clarification or description.
6.2: Evidence Selecting and using information to investigate a point of view or conclusion
Information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis.
Information is taken from source(s) with enough interpretation/evaluation to develop a coherent analysis or synthesis.
Information is taken from source(s) with some interpretation/evaluation, but not enough to develop a coherent analysis or synthesis.
Information is taken from source(s) without any interpretation/evaluation.
Viewpoints of experts are taken as fact, without question.
6.3: Influence of context and assumptions
Thoroughly (systematically and methodically) analyzes own and others' assumptions and carefully evaluates the relevance of contexts when presenting a position.
Identifies own and others' assumptions and several relevant contexts when presenting a position.
Questions some assumptions. Identifies several relevant contexts when presenting a position. May be more aware of others' assumptions than one's own (or vice versa).
Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions).
Continue onto next slide 25

26. The VALUE Rubric for the Critical Thinking Skill
Criterion
Exceed Standards 4
Meet Standards 3
Meet Minimum Standards 2
Don’t Meet Standards 1
6.4: Student's position (perspective, thesis/hypothesis)
Specific position (perspective, thesis/hypothesis) is imaginative, taking into account the complexities of an issue.
Limits of position (perspective, thesis/hypothesis) are acknowledged.
Others' points of view are synthesized within position.
Specific position (perspective, thesis/hypothesis) takes into account the complexities of an issue. Others' points of view are acknowledged within position (perspective, thesis/hypothesis).
Specific position (perspective, thesis/hypothesis) acknowledges different sides of an issue.
Specific position (perspective, thesis/hypothesis) is stated, but is simplistic and obvious.
6.5: Conclusions and related outcomes (implications and consequences)
Conclusions and related outcomes (consequences and implications) are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order.
Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly.
Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly.
Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified.
An example of the Program Standard of Performance for the Critical Thinking PLO:
Eighty percent (80%) of our graduate students should achieve a score of at least 3 in all dimensions of the above rubric (eg. Soc. 200B) and 3.5 by the time of graduation (Soc.500).
The program standard of performance helps programs identify how well students perform within and across the program learning outcome (PLO). 26

27. Define Standard of Performance
Plan step 3
Expectations 2:
-Define Standard of Performance
Answers to Q2.2.a: Q2: Standards of Performance/Expectations: Eighty percent (80%) of our students will score 3.0 or above in all five dimensions of the VALUE rubric (by the time they graduate).

28. Outline for one PLO: The PLO and Its Definition (Q2.1 and Q2.1.1)
The Rubric(s) and Standard(s) of Performance/Expectations (Q2.2a)
The Direct Measure(s) (Q3.3.2)
The Data Table(s) (Q4.1)
The Assessment Plan = Action Plan (Q5.1.1-Q5.1.2) 28

29. Attach assignment instructions that students received
Questions from Annual Assessment Report Template: Q Please attach the assignment instructions that the students received to complete the assignment (Appendix I): Q Which of the following direct measures (key assignments, projects, portfolios, course work, student tests, etc.) were used? [Check all that apply] 1. Capstone projects (including theses, senior theses), courses, or experiences 2. Key assignments from required classes in the program 3. Key assignments from elective classes 4. Classroom based performance assessments such as simulations, comprehensive exams, critiques 5. External performance assessments such as internships or other community based projects 6. E-Portfolios 7. Other portfolios 8. Other measure. Specify:
Plan step 4
Methods and Measures
-Assignments
-Tests
-Projects
Back to Guide

30. Example assignment description
Answers to Q3.3.2: The key assignment for the iMET program assessment is the Action Research Report. iMET used this Action Research Report (Master’s Thesis) included in an ePortfolio as its direct measure to assess its critical thinking program learning outcome. This culminating experience report (the master thesis) includes the following tasks: 1. Designing and implementing a study using data collection tools that will allow the students to "show" the reader what happened during and as a result of the intervention. 2. Sorting through the findings after collecting the data, looking for data that reveal some information pertinent to the study. 3. Looking for relationships (patterns) between the data. These patterns emerge from a variety of sources such as things that have happened, things that students have observed, things that people have said, and things that students have measured. These are the findings (conclusions) of the study.

31. Is the annual assessment useful to you?
What do you want your students to know, to do, and to value? Are your students struggling with thesis or critical thinking? If yes, what do you plan to do?

32. Outline for one PLO: The PLO and Its Definition (Q2.1 and Q2.1.1)
The Rubric(s) and Standard(s) of Performance/Expectations (Q2.2a)
The Direct Measure(s) (Q3.3.2)
The Data Table(s) (Q4.1)
The Assessment Plan = Action Plan (Q5.1.1-Q5.1.2) 32

33. Think-Pair-Share: Data Presentation
What does the data table look like?
What should be included in the data table?

34. Describe results
Questions from the Annual Assessment Report Template: Q4.1. Please provide tables and/or graphs to summarize the assessment data, findings, and conclusions for the selected PLO in Q2.1 (see Appendix 12 in our Feedback Packet Example). Please do NOT include student names and other confidential information. This is going to be a PUBLIC document. Q4.3. For the selected PLO, the student performance: 1. Exceeded expectations/standards 2. Met expectations/standards 3. Partially met expectations/standards 4. Did not meet expectations/standards 5. No expectations or standards have been specified 6. Don’t know
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Back to Guide

35. Meet Minimum Standards
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Example Data Table
Five Criteria/outcomes
Exceed Standards
(4)
Meet
Standards
(3)
Meet Minimum Standards
(2)
Don’t Meet Standards
(1)
Total
(N=13)
6.1: Explanation of issues
38%
(n=5)
54%
(n=7) 0%
(n=0) 8%
(n=1)
(100%, N=13)
6.2: Evidence (Selecting and using information to investigate a point of view or conclusion)
15%
(n=2)
40%
(n=6)
21%
(n=3)
6.3: Influence of context and assumptions
41%
20%
6.4: Student's position (perspective, thesis/ hypothesis)
23%
6.5: Conclusions and related outcomes (Implications and consequences)
15% (n=2)
55%

36. Meet Minimum Standards
Example Data Table
Five Criteria/Outcomes
Exceed Standards
(4)
Meet Standards
(3)
Meet Minimum Standards
(2)
Don’t Meet Standards
(1)
Total
(N=13)
6.1: Explanation of issues
38%
(n=5)
54%
(n=7) 0%
(n=0) 8%
(n=1)
(100%, N=13)
6.2: Evidence (Selecting and using information to investigate a point of view or conclusion)
15%
(n=2)
40%
(n=6)
21%
(n=3)
6.3: Influence of context and assumptions
41%
20%
6.4: Student's position (perspective, thesis/ hypothesis)
23%
6.5: Conclusions and related outcomes (Implications and consequences)
15% (n=2)
55%
6.1: = 92% achieving 3.0 or higher.
6.2: = 55% not achieving 3.0 or higher.
6.3: = 56% not achieving 3.0 or higher.
6.4: = 77% achieving 3.0 or higher.
6.5: = 70% achieving 3.0 or higher.

37. Meet Minimum Standards
Criterion 
Exceed Standards 4
Meet Standards 3
Meet Minimum Standards 2
Don’t Meet Standards 1
6.1: Explanation of issues
Issue or problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding.
Issue or problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions.
Issue or problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown.
Issue or problem to be considered critically is stated without clarification or description.
6.2: Evidence (Selecting and using information to investigate a point of view or conclusion)
Information is taken from source(s) with enough interpretation or evaluation to develop a comprehensive analysis or synthesis.
Information is taken from source(s) with enough interpretation or evaluation to develop a coherent analysis or synthesis.
Information is taken from source(s) with some interpretation or evaluation, but not enough to develop a coherent analysis or synthesis.
Information is taken from source(s) without any interpretation or evaluation.
Viewpoints of experts are taken as fact, without question.
6.3: Influence of context and assumptions
Thoroughly (systematically and methodically) analyzes own and others' assumptions and carefully evaluates the relevance of contexts when presenting a position.
Identifies own and others' assumptions and several relevant contexts when presenting a position.
Questions some assumptions. Identifies several relevant contexts when presenting a position. May be more aware of others' assumptions than one's own (or vice versa).
Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions).
6.4: Student's position (perspective, thesis/ hypothesis)
Specific position (perspective, thesis or hypothesis) is imaginative, taking into account the complexities of an issue.
Limits of position (perspective, thesis/hypothesis) are acknowledged.
Others' points of view are synthesized within position.
Specific position (perspective, thesis or hypothesis) takes into account the complexities of an issue. Others' points of view are acknowledged within position (perspective, thesis/hypothesis).
Specific position (perspective, thesis or hypothesis) acknowledges different sides of an issue.
Specific position (perspective, thesis or hypothesis) is stated, but is simplistic and obvious.
6.5: Conclusions and related outcomes (Implications and consequences)
Conclusions and related outcomes (consequences and implications) are logical and reflect students’ informed evaluation and ability to place evidence and perspectives discussed in priority order.
Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. 
Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. 
Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified.
Q2: Standards of Performance/Expectations: Seventy (70%) of our students will score 3.0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program.)

38. Results: Numerical Conclusions
Q2: Standards of Performance/Expectations: Seventy percent (70%) of our students will score 3.0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program.)
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Summary conclusion for Q4.2
Students meet the following standards 6.1 (92%), 6.4 (77%) and 6.5 (70%).
Students do not meet the following standards 6.2 (61%) and 6.3 (61%).
The two areas needing improvement:
6.2: Evidence (61%)
6.3: Influence of context and assumptions (61%).
Student Performance (Q4.3): 3. Partially meet the standards

39. Data Collection Sheet for Each Student
Criterion 
Exceed Standards 4
Meet Standards 3
Meet Minimum Standards 2
Don’t Meet Standards 1
6.1: Explanation of issues
Issue or problem to be considered critically is stated clearly and described comprehensively, delivering all relevant information necessary for full understanding.
Issue or problem to be considered critically is stated, described, and clarified so that understanding is not seriously impeded by omissions.
Issue or problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown.
Issue or problem to be considered critically is stated without clarification or description.
6.2: Evidence (Selecting and using information to investigate a point of view or conclusion)
Information is taken from source(s) with enough interpretation or evaluation to develop a comprehensive analysis or synthesis.
Information is taken from source(s) with enough interpretation or evaluation to develop a coherent analysis or synthesis.
Information is taken from source(s) with some interpretation or evaluation, but not enough to develop a coherent analysis or synthesis.
Information is taken from source(s) without any interpretation or evaluation.
Viewpoints of experts are taken as fact, without question.
6.3: Influence of context and assumptions
Thoroughly (systematically and methodically) analyzes own and others' assumptions and carefully evaluates the relevance of contexts when presenting a position.
Identifies own and others' assumptions and several relevant contexts when presenting a position.
Questions some assumptions. Identifies several relevant contexts when presenting a position. May be more aware of others' assumptions than one's own (or vice versa).
Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions).
6.4: Student's position (perspective, thesis/ hypothesis)
Specific position (perspective, thesis or hypothesis) is imaginative, taking into account the complexities of an issue.
Limits of position (perspective, thesis/hypothesis) are acknowledged.
Others' points of view are synthesized within position.
Specific position (perspective, thesis or hypothesis) takes into account the complexities of an issue. Others' points of view are acknowledged within position (perspective, thesis/hypothesis).
Specific position (perspective, thesis or hypothesis) acknowledges different sides of an issue.
Specific position (perspective, thesis or hypothesis) is stated, but is simplistic and obvious.
6.5: Conclusions and related outcomes (Implications and consequences)
Conclusions and related outcomes (consequences and implications) are logical and reflect students’ informed evaluation and ability to place evidence and perspectives discussed in priority order.
Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly. 
Conclusion is logically tied to information (because information is chosen to fit the desired conclusion); some related outcomes (consequences and implications) are identified clearly. 
Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified.
Q2: Standards of Performance/Expectations: Seventy percent (70%) of our students will score 3.0 or above in all the five dimensions of the VALUE rubric (by the time they graduate from the four semester program.)

40. Question 4 (Q4): An example of a Data Collection Sheet for Each Student/Assignment
Reference: Your data tables are based on the rubric and the data collection sheet.
The moment your rubric is developed, you also have a data collection sheet!
The following table is an example of a data collection sheet for a student:

41. Results: Conclusions Answer for Q4.2: Report step 1
-Data Tables
-Findings
-Conclusions
Results: Conclusions
Answer for Q4.2:
We can see (using the above table) that students meet the criteria for 6.1 (92%), 6.4 (77%), and 6.5 (70%) based on the assessment of our selected Critical Thinking PLO and our identified program standards of performance (70% of students should achieve a score of 3 or higher in all dimensions of the Critical Thinking Rubric). Students do not meet the criteria of 6.2 (61%) and 6.3 (61%). Students meet some of our program standards for the Critical Thinking Skill, thus they “Partially Met Program Standards.” Two areas need improvement: 1) Criterion 6.2: Evidence (61%), and 2) Criterion 6.3: Influence of context and assumptions (61%).

42. Outline for one PLO: The PLO and Its Definition (Q2.1 and Q2.1.1)
The Rubric(s) and Standard(s) of Performance/Expectations (Q2.2a)
The Direct Measure(s) (Q3.3.2)
The Data Table(s) (Q4.1)
The Assessment Plan = Action Plan (Q5.1.1-Q5.1.2) 42

43. Think-Pair-Share: Data Analysis and Presentation
How useful is the data?
Can you use the data to improve student learning and success?
What issues are you struggling with?
Are you collecting data to address those issues?
Have you used the data to promote your program?

44. Question 5.2: Use of Assessment Data
Q5.2. To what extent did you apply previous assessment results collected through your program in the following areas? [Check all that apply]
(1)
Very Much
(2)
Quite a Bit
(3)
Some
(4)
Not at all
(8)
N/A
1. Improved specific courses X
2. Modified curriculum
3. Improved advising and mentoring
4. Revised learning outcomes/goals
5. Revised rubrics and/or expectations
6. Developed/updated assessment plan
7. Annual assessment reports
8. Program review
9. Prospective student and family information
10. Alumni communication
11. WASC accreditation (regional accreditation)
12. Program accreditation x
13. External accountability reporting requirement
14. Trustee/Governing Board deliberations
15. Strategic planning
16. Institutional benchmarking
17. Academic policy development or modification
18. Institutional Improvement
19. Resource allocation and budgeting
20. New faculty hiring
21. Professional development for faculty and staff
22. Recruitment of new students
23. Other Specify:      
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop

45. Question 5.1.1: Use of Assessment Data
Questions from Annual Assessment Report Template: Q Please describe what changes you plan to make in your program as a result of your assessment of this PLO. Include a description of how you plan to assess the impact of these changes. Answer to Q5.1.1: In order to help students in our program successfully become critical thinking researchers, we will design more classroom activities and assignments related to: 1) Re-examination of evidence (6.2) and context and assumptions (6.3) in the research, and 2) Require students to apply these skills as they compose comprehensive responses for all their assignments.
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop
Back to Guide

46. Thank you!

47. A Simple Example 1
Educational Technology (iMET), MA (Example of a graduate-level Intellectual skills PLO with multiple dimensions)

48. A Simple Example 2
Chemistry, BS/BA (Example of an Undergraduate-level Disciplinary Competence PLO)

49. Other than GPA, What Data Are Used to Measure PLOs
Table 5: Sociology Evidence Map at the Program Level:
Other than GPA, What Data Are Used to Measure PLOs
Outcomes
Year
Outcome 1:
Advanced Sociological knowledge
Outcome 2:
Sociological contributions and applications
Outcome 3:
Challenges in sociology
Outcome 4:
Oral
Communication
Outcome 5:
Written
Outcome 6:
Critical
Thinking
Outcome 7:
Intercultural Knowledge & Competence
Outcome 8: Integration: Inquiry and Analysis
2012 – 2013
(program review)
Thesis
Research paper
Oral exam (thesis hearing)?
Exit survey?
Thesis defense
Self-study

50. How programs have used and/or can use assessment data
Q1.1.
Q5.1.
Very much
Quite a bit
Some
Not at all
No answer
Annual assessment reports
30%
12%
14% 8%
37%
Developing/updating assessment plan
21% 9%
36%
Revising rubrics and/or expectations 4%
Improving specific courses
19%
10%
25%
15%
31%
Program review
54%
Modifying curriculum
35%
13%
33%
Improving advising and mentoring 6%
26%
11%
44%
Revising learning outcomes/goals
23%
39%
Professional development for faculty and staff 5%
New faculty hiring
24%
51%
Academic policy development or modification
27%
53%
External accountability reporting requirement
22%
65%
Strategic planning
17%
46%
Alumni communication 1%
Program accreditation 3%
Prospective student and family information
Institutional Improvement
60%
WASC accreditation (regional accreditation) 0%
56%
Recruitment of new students
32%
Resource allocation and budgeting
Institutional benchmarking
67%
Trustee/Governing Board deliberations
18%
77%
36%
Written communication
32%
Inquiry and analysis
26%
Critical thinking
19%
Overall competencies in the major/discipline
Information literacy
18%
Oral communication
17%
Integrative and applied learning
13%
Ethical reasoning
10%
Problem solving 8%
Intercultural knowledge and competency
Others 6%
Quantitative literacy
Team work 5%
Creative thinking
Foundations and skills for lifelong learning 4%
Reading
Overall competencies for GE Knowledge 3%
Global learning 1%
Civic knowledge and engagement
The Office of Academic Program Assessment Preliminary Data, 2014

51. University Learning Outcomes
Baccalaureate Learning Goals
Graduate Learning Goals
AAC&U VALUE Rubrics
DQP
Bloom’s Revised Taxonomy

53. Review 2017-2018 Assessment Report Feedback Activity
Please review Assessment Report Feedback from this link.
Write the key commendation or recommendation for each component of the Assessment Report on the table we provide.
Plan step 1
PLOs:
-Select
-Define
-Curriculum map
Plan step 2
Expectations 1:
-Develop or adopt Rubrics
2. Definne
Plan step 3
Expectations 2:
-Define Standard of Performance
Plan step 4
Methods and Measures
-Assignments
-Tests
-Projects
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop

54. Sacramento State Baccalaureate Learning Goals
1. Competency in the Disciplines (to know)
2. Knowledge of Human Cultures and the Physical and Natural World (GE) (to know)
3. Intellectual and Practical Skills (to know and to do)
Personal and Social Responsibility (to value)
Integrative Learning (to do)

55. Graduate Learning Goals
Competency in the Discipline: Disciplinary Knowledge: Master, integrate, and apply disciplinary knowledge and skills to current, practical, and important contexts and situations. Intellectual and Practical Skills: Communication: Communicate key knowledge with clarity and purpose both within the discipline and in broader contexts. Critical Thinking/Analysis: Demonstrate the ability to be creative, analytical, and critical thinkers. Information Literacy: Demonstrate the ability to obtain, assess, and analyze information from a myriad of sources. Research (Doctoral Programs) Personal and Social Responsibilities: Professionalism: Demonstrate an understanding of professional integrity. Intercultural/Global Perspectives: Demonstrate relevant knowledge and application of intercultural and/or global perspectives.

56. AAC&U and VALUE Rubrics
3.1 Critical Thinking (WSCUC core competency)
3.2 Information Literacy (WSCUC core competency)
3.3 Written Communication (WSCUC core competency)
3.4 Oral Communication (WSCUC core competency)
3.5 Quantitative Literacy (WSCUC core competency)
3.6 Inquiry and Analysis (Sixth VALUE rubric)
3.7 Creative Thinking (Seventh VALUE rubric)
3.8 Reading (Eighth VALUE rubric)
3.9 Teamwork (Ninth VALUE rubric)
3.10 Problem Solving (Tenth VALUE rubric)

57. Appendices 8-9 Baccalaureate Learning Goals, Graduate Learning Goals, and AAC&U VALUE Rubrics (Detailed)
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
1.3 Intellectual and Practical Skills
1.3.1 Communication
1.3.1a Written
1.3.1b Oral
1.3.2 Critical Thinking
1.3.3 Literacy
1.3.3a Information
1.3.3b Quantitative
1.3.4 Inquiry and Analysis
1.3.5 Creative Thinking
1.3.6 Reading
1.3.7 Teamwork
1.3.8 Problem Solving
2.3 Intellectual and Practical Skills
2.3.1 Communication
2.3.2 Critical thinking/analysis
2.3.3 Information literacy
3.3 Intellectual and Practical Skills
3.3.1 Communication
3.3.1a Written
3.3.1b Oral
3.3.2 Critical Thinking
3.3.3 Literacy
3.3.3a Information
3.3.3b Quantitative
3.3.4 Inquiry and Analysis
3.3.5 Creative Thinking
3.3.6 Reading
3.3.7 Teamwork
3.3.8 Problem Solving
1.4 Personal and Social Responsibility (Values)
1.4.1 Global Learning
1.4.2 Intercultural Knowledge and Competence
1.4.3 Civic Knowledge and Engagement
1.4.4 Ethical Reasoning
1.4.5 Foundations and Skills for Lifelong Learning
2.4 Personal and Social Responsibility (Values)
2.4.1 Intercultural/global perspectives
2.4.6 Professionalism
3.4 Personal and Social Responsibility (Values)
3.4.1 Global Learning
3.4.2 Intercultural Knowledge and Competence
3.4.3 Civic Knowledge and Engagement
3.4.4 Ethical Reasoning
3.4.5 Foundations and Skills for Lifelong Learning
1.5 Integrative Learning
2.5 Integrative Learning
3.5 Integrative Learning
2.6 Research

58. Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, and DQP (Simple)
Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics and DQP.
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees)
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
4.1 Specialized Knowledge
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
4.2 Broad and Integrative Knowledge
1.3 Intellectual and Practical Skills
2.3 Intellectual and Practical Skills
3.3 Intellectual and Practical Skills
4.3 Intellectual Skills
1.4 Personal and Social Responsibility (Values)
2.4 Personal and Social Responsibility (Values)
3.4 Personal and Social Responsibility (Values)
1.5 Integrative Learning
2.5 Integrative Learning
3.5 Integrative Learning
4.5 Applied and Collaborative Learning
2.6 Research
4.6 Civic and Global Learning

59. Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, DQP, and Bloom’s Taxonomy (Simple)
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees)
5. Bloom’s Revised Taxonomy
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
4.1 Specialized Knowledge
5.1 Bloom’s Revised Taxonomy
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
4.2 Broad and Integrative Knowledge
5.2 Bloom’s Revised Taxonomy
1.3 Intellectual and Practical Skills
2.3 Intellectual and Practical Skills
3.3 Intellectual and Practical Skills
4.3 Intellectual Skills
5.3 AAC&U VALUE Rubrics (Valid Assessment of Learning in Undergraduate Education)
1.4 Personal and Social Responsibility (Values)
2.4 Personal and Social Responsibility (Values)
3.4 Personal and Social Responsibility (Values)
5.4 AAC&U VALUE Rubrics
1.5 Integrative Learning
2.5 Integrative Learning
3.5 Integrative Learning
4.5 Applied and Collaborative Learning
5.5 AAC&U VALUE Rubrics
2.6 Research
4.6 Civic and Global Learning
5.6 Civic and Global Learning

60. Baccalaureate Learning Goals, Graduate Learning Goals, AAC&U VALUE Rubrics, and DQP (Detailed)
Refer to the table below to know the relationship between the BLGs and GLGs and how they both relate to the AAC&U VALUE rubrics and DQP.
1. Baccalaureate Learning Goals (BLGs) (Appendix 8)
2. Graduate Learning Goals (GLGs) (Appendix 9)
3. AAC&U (Associations of American Colleges and Universities) VALUE Rubrics
4. DQP (Degree Qualification Profile: Learning Goals for Associate, Bachelor, and Master Degrees)
1.1 Competence in the Disciplines
2.1 Disciplinary Knowledge
3.1 Competence in the Disciplines
4.1 Specialized Knowledge
1.2 Knowledge of Human Cultures and the Physical and Natural World (GE)
3.2 Knowledge of Human Cultures and the Physical and Natural World
4.2 Broad and Integrative Knowledge
1.3 Intellectual and Practical Skills
1.3.1 Communication (Written and Oral)
1.3.2 Critical Thinking
1.3.3 Literacy
1.3.3a Information
1.3.3b Quantitative
1.3.4 Inquiry and Analysis
1.3.5 Creative Thinking
1.3.6 Reading
1.3.7 Teamwork
1.3.8 Problem Solving
2.3 Intellectual and Practical Skills
2.3.1 Communication
2.3.2 Critical thinking/analysis
2.3.3 Information literacy
3.3 Intellectual and Practical Skills
3.3.1 Communication (Written and Oral)
3.3.2 Critical Thinking
3.3.3 Literacy
3.3.3a Information
3.3.3b Quantitative
3.3.4 Inquiry and Analysis
3.3.5 Creative Thinking
3.3.6 Reading
3.3.7 Teamwork
3.3.8 Problem Solving
4.3 Intellectual Skills
1.4 Personal and Social Responsibility (Values)
1.4.1 Global Learning
1.4.2 Intercultural Knowledge and Competence
1.4.3 Civic Knowledge and Engagement
1.4.4 Ethical Reasoning
1.4.5 Foundations and Skills for Lifelong Learning
2.4 Personal and Social Responsibility (Values)
2.4.1 Intercultural/global perspectives
2.4.6 Professionalism
3.4 Personal and Social Responsibility (Values)
3.4.1 Global Learning
3.4.2 Intercultural Knowledge and Competence
3.4.3 Civic Knowledge and Engagement
3.4.4 Ethical Reasoning
3.4.5 Foundations and Skills for Lifelong Learning
1.5 Integrative Learning
2.5 Integrative Learning
3.5 Integrative Learning
4.5 Applied and Collaborative Learning
2.6 Research
4.6 Civic and Global Learning

61. Initial Emerging Developed Highly Developed
Appendix 3: WSCUC “Rubric for Assessing the Quality of Academic Program Learning Outcomes”
Criterion
Initial
Emerging
Developed
Highly Developed
1. Comprehensive List
The list of outcomes is problematic: e.g., very incomplete, overly detailed, inappropriate, and disorganized. It may include only discipline-specific learning, ignoring relevant institution-wide learning. The list may confuse learning processes (e.g., doing an internship) with learning outcomes (e.g., application of theory to real-world problems).
The list includes reasonable outcomes but does not specify expectations for the program as a whole. Relevant institution-wide learning outcomes and/or national disciplinary standards may be ignored. Distinctions between expectations for undergraduate and graduate programs may be unclear.
The list is a well-organized set of reasonable outcomes that focus on the key knowledge, skills, and values students learn in the program. It includes relevant institution-wide outcomes (e.g., communication or critical thinking skills). Outcomes are appropriate for the level (undergraduate vs. graduate); national disciplinary standards have been considered.
The list is reasonable, appropriate, and comprehensive, with clear distinctions between undergraduate and graduate expectations, if applicable. National disciplinary standards have been considered. Faculty has agreed on explicit criteria for assessing students’ level of mastery of each outcome.
2. Assessable Outcomes
Outcomes statements do not identify what students can do to demonstrate learning. “Statements understand scientific method” do not specify how understanding can be demonstrated and assessed.
Most of the outcomes indicate how students can demonstrate their learning.
Each outcome describes how students can demonstrate learning, e.g., “Graduates can write reports in APA style” or “Graduate can make original contributions to biological knowledge.”
Outcomes describe how students can demonstrate their learning. Faculty has agreed on explicit criteria statements such as rubrics, and have identified examples of student performance at varying levels of each outcome.
3. Alignment
There is no clear relationship between the outcomes and the curriculum that students experience.
Students appear to be given reasonable opportunities to develop the outcomes in the required curriculum.
The curriculum is designed to provide opportunities for students to learn and to develop increasing sophistication with respect to each outcome. This design may be summarized in a curriculum map.
Pedagogy, grading, the curriculum, relevant student support services, and co- curriculum are explicitly and intentionally aligned with each outcome. Curriculum map indicates increasing levels of proficiency.
4. Assessment Planning
There is no formal plan for assessing each outcome.
The program relies on short-term planning, such as selecting which outcome(s) to assess in current year.
The program has a reasonable, multi-year assessment plan that identifies when each outcome will be assessed. The plan may explicitly include analysis and implementation of improvements.
The program has a fully-articulated, sustainable, multi-year assessment plan that describes when and how each outcome will be assessed and how improvements based on findings will be implemented. The plan is routinely examined and revised, as needed.
5. The Student Experience
Students know little or nothing about the overall outcomes of the program. Communication of outcomes to students, e.g., in syllabi or catalog, is spotty or nonexistent.
Students have some knowledge of program outcomes. Communication is occasional and informal, left to individual faculty or advisors.
Students have a good grasp of program outcomes. They may use them to guide their own learning. Outcomes are included in most syllabi and are readily available in the catalog, on the web page, and elsewhere.
Students are well-acquainted with program outcomes and may participate in creation and use of rubrics. They are skilled at self-assessing in relation to the outcome levels of performance. Program policy calls for inclusion of outcomes in all course syllabi, and they are readily available in other program documents.

62. We have used the AAC&U VALUE rubric to organize the previous slide.

63. Question one! Let’s look at each of the components of a good assessment report!

64. Select PLOs/Competencies
Plan step 1
Critical Thinking
Information Literacy
Written Communication
Oral Communication
Quantitative Literacy
Inquiry and Analysis
Creative Thinking
Reading
Teamwork
Problem Solving
PLOs:
-Select

65. Q1.1. Which of the following Program Learning Outcomes (PLOs) … did you assess? [Check all that apply]
Critical Thinking (WASC) (GLG)
Ethical Reasoning
Information Literacy (WASC) (GLG)
Foundations and Skills for Lifelong Learning
Written Communication (WASC) (GLG)
Global Learning and Perspectives (GLG)
Oral Communication (WASC) (GLG)
Integrative and Applied Learning
Quantitative Literacy (WASC)
Overall Competencies for GE Knowledge
Inquiry and Analysis (GLG)
Overall Disciplinary Knowledge (GLG)
Creative Thinking
Professionalism (GLG)
Reading
Other
Team Work
Check here if your program has not collected any data for any PLOs.
Problem Solving
Civic Knowledge and Engagement
Intercultural Knowledge, Competency, and Perspectives (GLG)

66. Assessment Principles Assessment—an ongoing, interactive process using backward design and five basic principles:
Define Learning Goals
Outcomes & Definitions
Curriculum Map
Design Instruction
Help Students Achieve Outcomes
Decide on Assessments
Evidence of Learning Outcomes
“Backward Design” for Assessment (Outcomes & Definitions)
Plan step 1
PLOs:
-Select
-Define
-Curriculum map
Plan step 2
Expectations 1:
-Develop or adopt Rubrics
Plan step 3
Expectations 2:
-Define Standard of Performance
Plan step 4
Methods and Measures
-Assignments
-Tests
-Projects
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop
Q 2.1, Q21.1
Q 2.3
Q 2.2.a.
Q 3.3.2
Q 4.1-Q4.3
Q 20.2

67. Revised Bloom’s Taxonomy Table (2001)
Cognitive Dimension
Knowledge Dimension
1. Remember
1.1 Recognizing
1.2 Recalling
2. Understand
2.1 Interpreting
2.2 Exemplifying
2.3 Classifying
2.4 Summarizing
2.5 Inferring
2.6 Comparing
2.7 Explaining
3. Apply
3.1 Executing
3.2 Implementing
4. Analyze
4.1 Differentiating
4.2 Organizing
4.3 Attributing
5. Evaluate
5.1 Checking
5.2 Critiquing
6. Create
6.1 Generating
6.2 Planning
6.3 Producing
A. Factual Knowledge
Aa. of terminology
Ab. of specific details and elements
B. Conceptual Knowledge
Ba. of classifications and categories
Bb. of principles and generalizations
Bc. of theories, models, and structures
C. Procedural Knowledge
Ca. of subject-specific skills and algorithms
Cb. of subject-specific techniques and methods
Cc. of criteria for determining when to use appropriate procedures
D. Metacognitive Knowledge
Da. Strategic knowledge
Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge
Dc. Self-knowledge

68. Assignment 2
1.1: Can you use verbs to define what is critical thinking for your field of study and then place them in the Bloom’s table? 1.2: What kind of gap do you see in the teaching, learning, and assessment in the program?

69. PLO 6: Critical Thinking in the Sociology Graduate Program
6: Sociology graduate students will demonstrate a habit of systematically exploring social issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion” (Learning Goal/Outcomes): they will (PLO 6: Critical thinking adopted from the VALUE rubric):
6.1: Clearly identify and state the social issue/problem that needs to be considered critically, comprehensively describe the social issue/problem, and deliver all relevant information so it is necessary for a full understanding of the issue/problem (Explanation of issues);
6.2: Thoroughly interpret and evaluate sociological perspectives, theories, social methods/statistics, and any other current, credible, and relevant information (research, knowledge, tools, and/or views) to complete a thorough review of relevant literature and the problems to develop a comprehensive analysis or synthesis (Evidence);
6.3: Thoroughly analyze their own and others' assumptions and carefully evaluate the relevance of contexts when presenting a position (Influence of context and assumptions);
6.4: Consider the complexities (all sides) of a social issue. Limits of position and others' points of view are acknowledged and synthesized within position (Student's position);
6.5: Form conclusions, consequences and implications that are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order (Conclusions and related outcomes).

70. Revised Bloom’s Taxonomy Table (2001)
Cognitive Dimension
Knowledge Dimension
1. Remember
1.1 Recognizing
1.2 Recalling
2. Understand
2.1 Interpreting
2.2 Exemplifying
2.3 Classifying
2.4 Summarizing
2.5 Inferring
2.6 Comparing
2.7 Explaining
3. Apply
3.1 Executing
3.2 Implementing
4. Analyze
4.1 Differentiating
4.2 Organizing
4.3 Attributing
5. Evaluate
5.1 Checking
5.2 Critiquing
6. Create
6.1 Generating
6.2 Planning
6.3 Producing
A. Factual Knowledge
Aa. of terminology
Ab. of specific details and elements
6.1
6.2
6.5
B. Conceptual Knowledge
Ba. of classifications and categories
Bb. of principles and generalizations
Bc. of theories, models, and structures
C. Procedural Knowledge
Ca. of subject-specific skills and algorithms
Cb. of subject-specific techniques and methods
Cc. of criteria for determining when to use appropriate procedures
D. Metacognitive Knowledge
Da. Strategic knowledge
Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge
Dc. Self-knowledge
6.3/6.4

71. An Example: The Curriculum Map for the Sociology Graduate Program: Aligning (Linking) Graduate Program Learning Outcomes to Each Course in the Curriculum 1 “I” stands for “Introduced”, “D” for “Developed”, and “M” for “Mastered”.
Outcomes
Classes
Outcome 1:
Advanced Sociological Knowledge
Outcome 2:
Sociological Contributions and Applications
Outcome 3:
Challenges in Sociology
Outcome 4:
Oral
Communication
Outcome 5:
Written
Outcome 6:
Critical
Thinking
Outcome 7:
Intercultural Knowledge & Competence
Outcome 8: Integration: Inquiry and Analysis
Required Classes
SOC 200A (Introduction)
I 1 I
SOC 200B (Thesis Prep.)
D,M
SOC 214 (Methods) D
I, D
SOC 215 (Analysis)
SOC 235 (Soc. Psych.)
SOC 240 (Theory)
SOC 500/599 (Thesis Work) M
Elective Classes
SOC 210 (Urban Sociology)
SOC 220 (Change)
SOC 225 (Stratification)
SOC 226 (Gender)
SOC 230 (Social Org.)
SOC 238 (Environmental)
SOC 225 (Crime)
SOC 260 (Contem. Issues.)
SOC 265 (Race)
SOC 266 (Family)

72. Review 2017-2018 Assessment Report Feedback Activity (link)
Plan step 1
PLOs:
-Select
-Define
-Curriculum map
Plan step 2
Expectations 1:
-Develop or Adopt Rubrics
2. Define Standards of Performance
Plan step 3
Methods and Measures
-Assignments
-Tests
-Projects
Report step 1
Results
-Data Tables
-Findings
-Conclusions
Report step 2
Updated Assessment Action Plan:
-Using Assessment Data
-Closing the loop