VALUE/Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment Pilot Year Study Findings and Summary These slides summarize results from a proof-of-concept pilot study involving 59 institutions in nine states using common rubrics to assess more than 7,000 student work products. The sample of student work in the pilot represented the near-graduation students across the participating institutions in the nine states only; therefore, the results are not generalizable for all students in each participating state or nationwide.

MSC Pilot by the Numbers MSC states: Connecticut, Indiana, Kentucky, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, Utah 59 public institutions uploaded artifacts By sector: – 28 four-year, including 8 research institutions – 31 two-year These results are not generalizable across participating states or the nation in any way. Please use appropriately. 211/27/2015

MSC Pilot by the Numbers 7,215 pieces of student work were submitted [number of pieces of work approximates number of student participants] – Students had to be 75% of the way to completion of institutional degree requirements – 2,642 artifacts scored twice (36.6%) in order to measure inter-rater reliability 1,166 assignments were submitted [number of assignments approximates number of faculty participants] These results are not generalizable across participating states or the nation in any way. Please use appropriately. 311/27/2015

4 MSC Pilot Study Student Population Sample by Gender Relative to Graduating Students in Participating Institutions These results are not generalizable across participating states or the nation in any way. Please use appropriately. 11/27/2015

5 MSC Pilot Student Population of Pell-Eligible vs. Non Pell-Eligible Students These results are not generalizable across participating states or the nation in any way. Please use appropriately. 11/27/2015

6 MSC Pilot Study Four-Year Student Population Samples by Race 82% of students in MSC sample were White; 80% of students in participating institutions were White These results are not generalizable across participating states or the nation in any way. Please use appropriately. 11/27/2015

7 MSC Pilot Study Two-Year Student Population Samples by Race 77% of MSC sample were White students; 81% of students in participating institutions were White These results are not generalizable across participating states or the nation in any way. Please use appropriately. 11/27/2015

8 MSC Pilot Study Student Population Sample by Age Relative to Students at Participating Institutions These results are not generalizable across participating states or the nation in any way. Please use appropriately. 11/27/2015

VALUE Rubrics Valid Assessment of Learning in Undergraduate Education 11/27/20159 Assesses student work from curriculum and co-curriculum Provides detailed assessment of key dimensions/criteria of learning for each learning outcome Evaluated by faculty and other educational professionals Visible evidence of learning applied and used in multiple settings Shared framework and standards of judgment for quality without standardization

Faculty and Staff Responses to Usefulness of VALUE Rubrics for Assessing Student Work Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use 10 11/27/2015

Quantitative Literacy Rubric Dimensions Interpretation Ability to explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words) Representation Ability to convert relevant information into various mathematical forms (e.g., equations, graphs, diagrams, tables, words) Calculation Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized)

Note: Each work product was scored on 6 dimensions of quantitative literacy using a common AAC&U VALUE Rubric. See Slide 15 for rubric dimension criteria. VALUE rubrics are available at 12DRAFT11/27/2015 This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately.

Note: Each work product was scored on 6 dimensions of quantitative literacy using a common AAC&U VALUE Rubric. See Slide 15 below for rubric dimension criteria. VALUE rubrics are available at 13DRAFT11/27/2015 This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately.

Selected Findings – Quantitative Literacy Most students in both two-year and four- year institutions were rated “3” or “4” on students’ “calculation” skills. Less than half of work products at four-year institutions and about one-third of work products at two-year institutions were rated “3” or “4” on ability to “make judgments and draw appropriate conclusions based on quantitative analysis of data.”

Written Communication Rubric Dimensions Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s). Content Development Genre and Disciplinary Conventions Formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields (please see glossary). Sources and Evidence Control of Syntax and Mechanics

Note: Each work product was scored on 5 dimensions of written communication using a common AAC&U VALUE Rubric. See Slide 18 below for rubric dimension criteria. VALUE rubrics are available at 16DRAFT11/27/2015 This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately.

17DRAFT11/27/2015 Note: Each work product was scored on 5 dimensions of written communication using a common AAC&U VALUE Rubric. See Slide 18 below for rubric dimension criteria. VALUE rubrics are available at This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately.

Selected Findings – Written Communication Students at two-year colleges have students performing really well, for example: Nearly 50 percent of work products collected from two-year institutions were scored either “3” or “4” on “content development” in writing. About one-third of work products collected from two-year institutions were scored “3” or “4” on demonstrating the use of “sources and evidence” in writing. Students at four-year colleges and universities are performing at higher levels overall. More than 50 percent of student work at four-year institutions was scored at either a “3” or “4” overall in terms of written communication One still sees much room for improvement overall, especially in students’ capacity to use “sources and evidence” to make arguments in their writing.

Critical Thinking Rubric Dimensions Explanation of issues Evidence Selecting and using information to investigate a point of view or conclusion Influence of context and assumptions Student's position perspective, thesis/hypothesis Conclusions and related outcomes implications and consequences

Note: Each work product was scored on 5 dimensions of critical thinking using a common AAC&U VALUE Rubric. See Slide 12 for rubric dimension criteria. VALUE rubrics are available at 20DRAFT11/27/2015 This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately.

21DRAFT11/27/2015 This was a pilot study. These results are not generalizable across participating states or the nation in any way. Please use appropriately. Note: Each work product was scored on 5 dimensions of critical thinking using a common AAC&U VALUE Rubric. See Slide 12 for rubric dimension criteria. VALUE rubrics are available at

Selected Findings - Critical Thinking Consistent with other national studies, many students who had earned at least 75 percent of credits toward their degrees still are not achieving high levels of important critical thinking skills. Of the three outcomes evaluated, critical thinking was the worst. Less than one-third of student work products collected from four- year institutions were scored a “3” or “4” on “using evidence to investigate a point of view or reach a conclusion.” Nearly 40 percent of work samples assessed at four-year institutions were rated only “0” or “1” on how well students “analyzed the influence of context and assumptions” to draw conclusions. On the other hand, nearly 45 percent of work products from four- year institutions were rated “3” or “4” in terms of how well students “explained issues” in their analysis. In short, students can describe and explain, but lack proficiency in critically examining the assumptions, hypotheses, evidence and implications of issues and positions, i.e. applying their knowledge.

VALUE/Multi-State Collaborative (MSC) KEY CONCLUSIONS  Central to student achievement is the quality of assignments that will allow us to reap the evidence on learning  Faculty collaboration and agreement on quality of student work across and within institutions is possible and assessment results can be useful for improvement of learning  We can and need to improve what we are doing to achieve heightened proficiency among all students 11/27/