The assessment process Steve Culver Director, Assessment and Analytics Summer 2017 College & Department Workshop

WHAT IS ASSESSMENT? WHAT IS ASSESSMENT? A systematic processes for determining whether or not a program is meeting its outcomes and then utilizing this information to make program decisions. Grading is not necessarily assessment. Grades are related to achievement in a class; assessment demonstrates achievement related to outcomes of the program. Grades need to be broken down into learning outcome components to be used for assessment purposes.

WHY DO ASSESSMENT? WHY ASSESSMENT? Identify strengths and areas for improvement Provide student learning evidence for stakeholders, including discipline-specific and regional accreditors Highlight program contributions to stakeholders Encourage collaboration among program faculty Create a program vision and ideal Encourage curriculum review BUT the PRIMARY REASON is to improve the teaching and learning process

STEPS IN THE PROCESS: WHAT IS ASSESSMENT? “establishing clear, measurable expected outcomes of student learning; ensuring that students have sufficient opportunities to achieve those outcomes; systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches our expectations; and using the resulting information to understand and improve student learning.” Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco: Jossey-Bass

WHAT IS ASSESSMENT?

WHAT ARE THE SACSCOC REQUIREMENTS? The Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) Comprehensive Standard: “3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: 3.3.1.1 educational programs, to include student learning outcomes” (p.27). Educational programs implies any credentialed program including certificates Southern Association of Colleges and Schools Commission on Colleges (2011, December, p. 27). The principles of accreditation: Foundations for quality enhancement. Retrieved from http://www.sacscoc.org/pdf/2012PrinciplesOfAcreditation.pdf

WHAT DOES SACSCOC CONSIDER WHEN EXAMINING THIS STANDARD? “How are expected outcomes clearly defined in measurable terms for each educational program? What assessment instruments are used and why were they selected? Have the programs assessed the extent to which they have been successful in achieving their learning outcomes? If called for, have program improvements been made as a result of assessment findings?” Southern Association of Colleges and Schools Commission on Colleges (Second Edition 2012, p.55). Resource manual for the principles of accreditation: Foundations for quality enhancement. Retrieved from http://www.sacscoc.org/pdf/Resource%20Manual.pdf

STARTS WITH THE MISSION STATEMENT A concise statement about the values and purpose of the program The foundation a program uses to help define its program goals and outcomes Consistent with institutional and college mission Structure of a program mission statement: “The mission of (name of your program or unit) is to (your primary purpose) by providing (your primary functions or activities) to (your stakeholders).” Additional clarifying statements.

STUDENT LEARNING OUTCOMES PROGRAM & STUDENT LEARNING OUTCOMES PROGRAM OUTCOMES Reflects the services the program provides or student achievement in program areas (e.g., retention, graduation rates, employment statistics) Example: Full-time students who declare XYZ as a major will complete the program in 4 years. STUDENT LEARNING OUTCOMES Reflects the knowledge, skills, abilities that students are expected to acquire as a result of being in the program. Example: Graduates should be able to explain how past events in the area of XYZ impact society now.

A GOOD PROGRAM OUTCOME: PROGRAM & STUDENT LEARNING OUTCOMES A GOOD PROGRAM OUTCOME: Defines program services Is meaningful to the program Focuses on what your programs wants to accomplish or a service it provides Specifies program student achievement areas Can be reliably and validly measured A specific action or achievement related to program mission that can be improved upon Examples: Within two years of graduating from the program, students will obtain licensure. The department of XYZ will increase its graduation rates. The MA program will develop new courses to meet discipline-specific accreditation requirements.

PROGRAM & STUDENT LEARNING OUTCOMES At A&T, the University’s goal is that all degree programs will achieve excellence and recognition for high quality teaching and learning, including the achievement of a satisfactory external review and/or accreditation: At least one program outcome should be in relation to this goal. Strategic Plan 2011-2020 A&T Preeminence 2020 Embracing Our Past, Creating Our Future

A GOOD STUDENT LEARNING OUTCOME: PROGRAM & STUDENT LEARNING OUTCOMES A GOOD STUDENT LEARNING OUTCOME: A specific behavior or knowledge students are expected to be able to demonstrate as a result of the program Focuses on what you want students to know or be able to do / narrowly focused, measurable Completes the statement: at the end of the program, students will be able to ….. Narrowly focused, measurable, not a “bundled” statement Appropriate cognitive level for the degree (Bloom’s taxonomy) Examples: Students will be able to compare and contrast the style and form used in Monet’s earlier and later works. Students will be able to explain the logic of the field-field approach to American anthropology.

PROGRAM & STUDENT LEARNING OUTCOMES At A&T, the University has adopted four goals, so programs should identify at least one learning outcome for each goal. Students completing this degree program will exhibit effective communication skills (written, oral, interpersonal) appropriate for professionals in this field of study. Students completing this degree program will effectively use quantitative and qualitative analytical problem-solving skills appropriate for professionals in this field of study. Students completing this degree program will demonstrate a level of discipline-specific expertise (knowledge, skills, and professionalism) appropriate for professionals in this field of study. Students completing this degree program will demonstrate ability to engage productively in the review and conduct of disciplinary research appropriate for professionals in this field of study.

Programs should have a mix of student learning and program outcomes: OUTCOMES—HOW MANY? Programs should have a mix of student learning and program outcomes: 4 to 7 student learning outcomes 2 to 3 program outcomes Programs with discipline-specific accreditation requirements (e.g., ABET, AACSB, etc.) might have more outcomes Each program should measure at least 2-3 student learning outcomes and 1-2 program outcomes every year. All outcomes should be measured at least twice in five years.

CURRICULUM MAPPING CURRICULUM MAPPING Allows programs to align learning outcomes with courses Exposes gaps in the curriculum Assists assessment planning by showing where and when assessment can be (may already be) implemented Improves communication among faculty Improves program coherence Increases likelihood that students will achieve outcomes

An example of what a curriculum map might look like CURRICULUM MAPPING An example of what a curriculum map might look like Student Learning Outcome (SLO) Required Introductory Courses (100 / 200 level) Required Upper Level Courses (300 /400 level) Concentration Specific Requirements Additional Required Academic Experience   BIOL 1006 BIOL 2010 BIOL 3025 BIOL 4030 BIOL 4064 BIOL 3018 Internship (required) SLO#1: Explain the core biological concepts related to evolution and principles of genetics. I R, A R SLO #2: Critically analyze biological research and findings. Continue with the rest of your CORE learning outcomes SLO Concentration Example: Explain the use of cells and biological materials in biochemistry. Complete for each concentration outcome “I” indicates where the concept is first being introduced to the students, “R” represents where the concept is being reinforced, and “A” indicates the assessment point.

WHAT TO CONSIDER WHEN CHOOSING A MEASURE MEASURES WHAT TO CONSIDER WHEN CHOOSING A MEASURE compatible with the outcomes provide reasonably accurate, useful information yield results specific enough to show where improvements could be made match the cognitive level of the outcome Find out what activities faculty are already doing in their courses Find out what is being done institution-wide that might map to your outcomes Utilize one activity for several outcomes when possible Use capstone points in the curriculum

BE CAREFUL WHAT YOU ASK FOR….. MEASURES BE CAREFUL WHAT YOU ASK FOR…..

DIRECT MEASURES – tangible, visible, observable Student learning outcomes: Student artifacts (e.g. capstone projects, portfolios, presentations, case studies, etc.) examined by learning outcome and scored with a rubric Externally reviewed exhibitions, performances, or projects scored by learning outcome External evaluation of performance during internships based on outcomes. Commercially-developed tests, locally-developed tests, national licensure exams or professional exams if examined by learning outcome Program outcomes: Number of students meeting certain criteria (e.g., % attending graduate school) or number of projects or initiatives implemented (e.g., courses developed, grants obtained, etc.) Enrollment numbers / number of majors

INDIRECT MEASURES – infer student knowledge, not directly observable Survey research: National or local instruments Exit surveys / senior surveys, alumni or employer surveys National Survey of Student Engagement (NSSE), The College Senior Survey (CSS), etc. Faculty surveys Course evaluations Employer satisfaction studies and advisory boards Exit interviews and student focus groups Self-assessments Peer ratings

METHODOLOGY MAPS CAN HELP WITH PLANNING: MEASURES METHODOLOGY MAPS CAN HELP WITH PLANNING: An example of what an assessment methodology map might look like Student Learning Outcome (SLO) Dept Direct Assessment Methods Dept Indirect Assessment Methods   Capstone course projects Written course assignments Standardized subject matter test Graduating senior survey Dept exit interviews Alumni survey SLO#1: Explain the core biological concepts related to evolution and principles of genetics. BIOL 4030 – collected by professor every time course is taught Conducted by the department every year Conducted by OSPIE every year Conducted by members of the curriculum committee every spring

(Decided to assess to determine if action plan is working) MEASURES TIMELINE MAPS CAN HELP WITH PLANNING: An example of what an assessment timeline map might look like Student Learning Outcome (SLO)   2015-2016 2016-2017 2017-2018 2018-2019 2019-2020 2020-2021 SLO#1: Explain the core biological concepts related to evolution and principles of genetics. BIOL capstone project Standardized test Standardize d test SLO#2: Critically analyze biological research and findings. (outcome not met) (Decided to assess to determine if action plan is working)

TARGETS WHAT IS A PERFORMANCE TARGET? Defines the program’s achievement expectations for each outcome-method pair Is usually expressed as percentages or numbers expected Is achievable but rigorous Is realistic for the program level Examples: Student learning outcome: 80% of students will meet or exceed expectations on the rubric items for this outcome. Program outcome: 80% of students who declare XYZ as a major will complete the program in 4 years.

ACTION PLANS WHAT MAKES A GOOD ACTION PLAN? Action plans: Describe the changes a program intends to implement to address curriculum or program deficiencies identified as a result of the assessment Describe specific programmatic changes with timelines if possible Should be discussed among the faculty to involve them in the improvement or planning process Changes to curriculum, assessment plan or methods, pedagogical practices, implementation of new technology or assignments

CLOSING THE LOOP SEVERAL WAYS TO “CLOSE THE LOOP” #1: Assessment results show that program has achieved its outcome. Data were collected, results show outcome achieved given the specific target; program plans to continue to measure outcome in future assessment cycles. #2: Assessment results show that program has not achieved its outcome. Data were collected, results show outcome not achieved given the specific target; program developed/implemented action plan to improve performance; program examined impact of action plan by re-measuring and collecting data on outcome in a future cycle. #3: Program has determined that the assessment plan is ineffective and needs to be revised. Data were collected; program determined that plan was not accurately measuring the outcome or that outcome needed revision; program documented changes to be made in the plan and re-measured in future cycle.

REPORTING

Taskstream as the repository for our work Reports due: Assessment plans for the year - August 25th Assessment results report - June 29th Two major areas in Taskstream: Standing requirements includes mission, program/student learning outcomes Annual Assessment Plan includes assessment results report and assessment plans for the year

Summary Summary and Review Mission statement Student learning and program outcomes Curriculum mapping Measures and Targets Action plans Documentation and Reporting