AAC&U is working in collaboration with Indiana University’s Center for Postsecondary Research (IUCPR) and the Multi-State Consortium of State Higher Education Executive Officers (SHEEO) to establish the VALUE Institute, which is a continuing resource for higher education institutions to document, report, and use learning outcomes evidence to improve student success in college. VALUE INSTITUTE Registration Now Open http://valueinstituteassessment.org

Use Q&A for: Panel discussion Use Chat for: Technology support #aacuVALUE

Speakers Moderator Terrel R. Rhodes Vice President, Office of Quality, Curriculum, and Assessment and Executive Director of VALUE  Association of American Colleges and Universities (AAC&U) Rhodes@aacu.org Thomas Nelson Laird Director, Indiana University’s Center for Postsecondary Research (IUCPR) tflaird@indiana.edu Robert J. Awkward Director of Learning Outcomes Assessment Massachusetts Department of Higher Education rawkward@dhe.mass.edu Cara Smith Director of Institutional Effectiveness Georgia College cara.smith@gcsu.edu

Terrel R. Rhodes Vice President, Office of Quality, Curriculum, and Assessment and Executive Director of VALUE  Association of American Colleges and Universities (AAC&U) Rhodes@aacu.org

Origin of AAC&U VALUE Rubrics Critical Thinking, Creative Thinking, Written Communication, Oral Communication, Quantitative Literacy, Information Literacy, Teamwork, Problem Solving, Civic Knowledge & Engagement, Intercultural Knowledge & Competence, Ethical Reasoning & Action, Global Learning, Lifelong Learning, Integrative Learning, Inquiry and Analysis, Reading 2007–2009 teams of faculty and other educational professionals developed 16 VALUE rubrics for the LEAP Essential Learning Outcomes. Each rubric was developed from core identified characteristics or criteria of learning for each outcome. Rubrics tested by faculty with their own students’ work. TR

VALUE Institute Assumptions Student work is representation of student motivated learning Focus on what student does in terms of key dimensions of learning outcomes Faculty and educator expert judgment Results are useful and actionable for improvement of learning Raise up, not wash out, inherent diversity—race, ethnicity, and socioeconomic status and diversity of courses, credit-levels, and disciplinary backgrounds—on campuses. JK

Where Authentic Measures of Student Learning Come From: Assignments and Artifacts Course-based assignments Papers, lab reports, performances General education outcome measures Writing-Intensive sequence artifacts Program-level (Major) capstone products Portfolios Cocurricular learning products/artifacts TR

Expert Faculty Scorers Scorer training sessions that are equal parts calibration to reach a consensus score and a rich faculty development opportunity, and that are open to all faculty whether they are contingent or tenure-track, two-year or four-year, curricular or cocurricular. TR

VALUE Rubrics and Equity The VALUE rubrics were purposefully designed to reflect an assets-based—versus deficit-focused—approach to assessing student learning (i.e., let’s focus on what students can do and build from that solid base). The rubric “descends” from the level-four Capstone to the level-one Benchmark when reading from left to right. Scorers are trained to assess student work by beginning at the highest levels of the rubric, working from the assumption that all students have the potential for achieving Capstone-level work. Scorers immediately orient themselves to the learning that is possible. TR Sullivan and McConnell in next issue of Change

VALUE Institute: Changing the Conversation about Learning Outcomes Assessment JK

Reasons for Participating in VALUE Institute Baylor: General Education assessment. Wanted to move from current end of program assessment to an Early and Late model and identify how these skills are developing. University of North Dakota: To gain an outside-of-the-institution scoring of student learning goals. Hoping to have the ability to benchmark student performance to better understanding students' strengths and weaknesses. JK

Thomas Nelson Laird Director, Indiana University’s Center for Postsecondary Research (IUCPR) tflaird@indiana.edu

2018 Spring Inaugural VALUE Institute: 26 Institutions 5 Outcomes >5,000 artifacts 127 scorers TNL

2018 Spring Inaugural VALUE Institute Participants Baylor University Bluegrass Community and Technical College Community College of Rhode Island Gateway Community College Georgia College Grand Canyon University Itasca Community College Ivy Tech Community College New College of Florida Norfolk State University North Shore Community College Northern Kentucky University Queen’s University Quinnipiac University Ryerson University Southeast Missouri State University Southern Connecticut State University Southern Oregon University State Fair Community College Suffolk University University of Hawaii at Manoa University of Kentucky University of Maine University of Massachusetts Amherst University of North Dakota Vincennes University

VALUE Institute: Learning Outcomes (100 artifacts per outcome) Outcome/Rubric Number of Institutions Written communication 20 Quantitative literacy 11 Critical thinking 22 Civic engagement Ethical reasoning Intercultural knowledge and competence Global learning 1 JK *

How Does the VALUE Institute Work? For each learning outcome selected, 100 artifacts can be scored Demographic information about the students and some information about the assignment/course/source of the work is also collected These data are combined for scoring and data analysis reporting The results are provided back to the submitting participant in reports and a database JK

Institutional Reports Critical Thinking results SAMPLE Results breakdown of student scores overall by rubric criteria and levels JK

Institutional Reports Critical Thinking results SAMPLE Results by Pell Eligibility JK

A Few Words on Scoring Each artifact scored by two trained, certified VALUE scorers VALUE Institute-certified score is the calculated average of the two scores, rounded up to the nearest whole number Artifacts that received a zero from either of the two scorers received a score of zero overall, as there was an absence of evidence of student learning on that dimension Artifacts for which the two scorers notably disagreed (in that the majority of dimensions received scores more than two apart) underwent a separate process -- a third score from an “expert” VALUE scorer. These triple-scored artifacts were then analyzed for patterns to determine the nature of the “true” score - averaged the two original scores and rounding up to the nearest whole number JK

Critical Thinking No Evidence 13% 21% 27% 15% 20%

Written Communication No Evidence 1% 2% 4% 14%

Robert J. Awkward Director of Learning Outcomes Assessment Massachusetts Department of Higher Education rawkward@dhe.mass.edu

Commonwealth of Massachusetts Public higher education is comprised of: Five University of Massachusetts campuses Nine state universities 15 community colleges

Student Learning Outcomes Assessment Brief History External factors Have faculty drive assessment Avoid standardized tests Address lack of assessment culture Create assessment initiative Working Group on Student Learning Outcomes and Assessment established the following in January 2010:

Massachusetts Organizing Structure Student Learning Outcomes Assessment Massachusetts Organizing Structure Massachusetts Task Force on Statewide Assessment Chair: Dr. Lane Glenn, President, Northern Essex Community College Providing broad oversight for Massachusetts State Level Amcoa Team Advancing assessment of student learning on public campuses; strengthening campus assessment capabilities Chair: R. J. Awkward VALUE Institute Steering Committee Representing MA in the VALUE Institute/MSC to develop a multistate program of assessment Chair: R. J. Awkward

How Did We Get Here? Massachusetts Pilot in 2012-2013 Student Learning Outcomes Assessment How Did We Get Here? Massachusetts Pilot in 2012-2013 Massachusetts joined with the Multi-State Collaborative (Mass. Pilot Phase II) in 2014-2015 AAC&U and SHEEO provided a common framework: Authentic Student Work LEAP VALUE Rubrics • Critical Thinking • Quantitative Literacy • Written Communication Common faculty training and scoring

Student Learning Outcomes Assessment How Did We Get Here? Multi-State Collaborative to Advance Quality Student Learning continued: Demonstration Year (2015-2016) Refinement Year (2016-2017) VALUE Institute Inaugural Year (2017-2018) Significant Learning Acquired as documented in “On Solid Ground” (VALUE Report, AAC&U, 2017)

Massachusetts Compared to Project Level Data VALUE Rubrics Two-Year Institutions Four-Year Institutions Critical Thinking + Written Communication +/- Quantitative Literacy -/+ -

Suggested Items for Individual Campuses & Statewide Action Written Communications More activities related to Written Communications in 3rd and 4th year courses, programs, and majors (except in Syntax and Mechanics) Resume or enhance existing “writing across the curriculum” activities Quantitative Literacy Employ quantitative literacy across the curriculum in addition to general education courses Increase activities related to Quantitative Literacy in 3rd and 4th year courses, programs, and majors Massachusetts Department of Higher Education DHE to provide regional assignment design workshops for faculty linked to VALUE Rubrics to improve teaching effectiveness, increase student learning, and improve outcomes results DHE to provide training for faculty on how to employ Quantitative Literacy in curriculum at all levels by launching regional Quantitative Literacy faculty workshops in 2018-2019

Cara Smith Director of Institutional Effectiveness Georgia College cara.smith@gcsu.edu

Georgia College & State University Located in rural Milledgeville, Georgia 4-year public, SACSCOC accredited, Master’s (larger programs) Fall 2018: 4,329 applications, 3,367 admitted, 1,483 enrolled Total enrollment: 6,989 Undergraduate: 5,958 Graduate: 1,031 Average SAT: 1,211 Average ACT: 25 17/1 student to faculty ratio

AAC&U ELO: Critical Thinking

GC Core Curriculum University System of Georgia (Areas A-F) Area B, “Institutional Options” GC1Y: Critical Thinking:_________________ Into the Wilderness, Ability/Disability, Social Problems, Music and Conflict, Utopia/Dystopia, Research in the Google Age, We the People, Growing up with Literature GC2Y: Global Perspectives: _________________ The AIDS Pandemic, Arts & Literacy, Big Data, Black Women Artists of the African Diaspora, Captivity and Freedom, Global Connections, History of Global Health

Assessing Area B Customized hybrid rubric aligned to: AAC&U Critical Thinking ELO rubric elements 1, 2 and 5 Area B core outcome (GC SLO 3) GC Critical thinking outcome, element 2 (GC SLO 13) However, faculty are supposed to utilize AAC&U Critical Thinking framing language, definition, and entire rubric when designing an Area B course

Administration Summary, Spring 2018 31 sections taught by 20 different professors 14 departments/disciplines represented Employed a simple random sample within courses, soliciting 5 samples per section Submitted 112 (out of a possible 155) artifacts from 14 unique assignments from 13 professors

Very positive for 1st year, first semester

How Will We Use Our Data? Through dialogue, validate faculty teaching and student learning Validate and strengthen internal assessment processes and practices (Spring 2019, Fall 2019) Refine assignment alignment Create faculty development opportunities Train new GC1Y instructors

Benefit for Georgia College and Beyond Benefit for Georgia College and Beyond? What was the Return on Our Investment? VERY validating for what we are already doing Inform future practice Increased appreciation and awareness of assessment Increased faculty buy-in Develop faculty leaders Coupled with this years’ participation, add to the national conversation of assessing high-impact practices (NASH TS3 HIPS Project)

JK http://valueinstituteassessment.org

Use Q&A for: Panel discussion Use Chat for: Technology support #aacuVALUE

Speakers Moderator Terrel R. Rhodes Vice President, Office of Quality, Curriculum, and Assessment and Executive Director of VALUE  Association of American Colleges and Universities (AAC&U) Rhodes@aacu.org Thomas Nelson Laird Director, Indiana University’s Center for Postsecondary Research (IUCPR) tflaird@indiana.edu Robert J. Awkward Director of Learning Outcomes Assessment Massachusetts Department of Higher Education rawkward@dhe.mass.edu Cara Smith Director of Institutional Effectiveness Georgia College cara.smith@gcsu.edu

Forthcoming AAC&U Webinars Wednesday, March 6, 2019 Truth, Racial Healing & Transformation Campus Centers Tuesday, April 16, 2019 Integrating Interfaith Engagement with Curricular and Cocurricular Initiatives