Elise Martin, Dean of Assessment, Middlesex Community College Cathy Pride, Assistant Professor, Psychology, Middlesex Community College Alice Frye, Lecturer, Psychology, University of Massachusetts, Lowell Assessing and Reporting Degree Qualifications Profile Competencies AAC&U General Education & Assessment Conference March 2, 2013

“Quality Collaboratives: Assessing and Reporting Degree Qualifications Profile (DQP) Competencies in the Context of Transfer” AAC&U project; support from Lumina Foundation Transfer partner teams (dyads) from 8 states, using Lumina’s DQP, develop educational practices to: help students achieve essential outcomes at appropriately high levels; document students’ attainment of outcomes; and facilitate students’ transfer of courses and competencies from two-year institutions to four-year institutions towards completing college degrees.

What is the DQP? “A framework for defining and ultimately measuring the general knowledge and skills that individual students need to acquire in order to earn degrees at various levels, such as associate, bachelor’s and master’s degrees. The Degree Profile is intended to help define generally what is expected of college graduates, regardless of their majors or fields of study. Lumina will fund experiments within a variety of settings.” (press release from Lumina Foundation, January 25, 2011) DQP Framework Categories: Specialized knowledge Broad, Integrative knowledge Intellectual skills (analytic inquiry, quantitative fluency, communication fluency, engaging diverse perspectives, use of information resources) Applied Learning Civic Learning

Lumina DQP – Quantitative Fluency Associate level: Student presents accurate calculations and symbolic operations, explains how such calculations and operations are used in specific field of study or in interpreting social and economic trends. Bachelor level: Student translates verbal problems into mathematical algorithms and constructs valid mathematical arguments using the accepted symbolic system of mathematical reasoning. Student constructs, as appropriate to his or her major field (or another field), accurate and relevant calculations, estimates, risk analyses or quantitative evaluations of public information and presents them in papers, projects or multi-media events.

Overview of UML/MCC Dyad Project Discipline-based teams from high transfer programs between institutions: Biology, Business, Psychology, and Criminal Justice Faculty from both institutions collaborate to develop contextualized curricula and assessments for QL for 200 and 400 level courses, using DQP benchmarks for Associate and Bachelor Degree student achievement mapped to VALUE Quantitative Literacy rubric Faculty from both institutions collaborate to apply the rubric to samples of student work generated by above curricula development VALUE QL rubric used formatively for curricula/assessment development and summatively to evaluate student learning

VALUE (LEAP ) Quantitative Literacy rubric Capstone 4 Milestones 32 1 Interpretation Ability to explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words) Provides accurate explanations of information presented in mathematical forms. Makes appropriate inferences based on that information. For example, accurately explains the trend data shown in a graph and makes reasonable predictions regarding what the data suggest about future events. Provides accurate explanations of information presented in mathematical forms. For instance, accurately explains the trend data shown in a graph. Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line. Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means. For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends. Representation Ability to convert relevant information into various mathematical forms (e.g., equations, graphs, diagrams, tables, words) Skillfully converts relevant information into an insightful mathematical portrayal in a way that contributes to a further or deeper understanding. Competently converts relevant information into an appropriate and desired mathematical portrayal. Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate. Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate. CalculationCalculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations are also presented elegantly (clearly, concisely, etc.) Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations attempted are either unsuccessful or represent only a portion of the calculations required to comprehensively solve the problem. Calculations are attempted but are both unsuccessful and are not comprehensive. Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work. Uses the quantitative analysis of data as the basis for competent judgments, drawing reasonable and appropriately qualified conclusions from this work. Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work. Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Explicitly describes assumptions and provides compelling rationale for why each assumption is appropriate. Shows awareness that confidence in final conclusions is limited by the accuracy of the assumptions. Explicitly describes assumptions and provides compelling rationale for why assumptions are appropriate. Explicitly describes assumptions.Attempts to describe assumptions. Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized) Uses quantitative information in connection with the argument or purpose of the work, presents it in an effective format, and explicates it with consistently high quality. Uses quantitative information in connection with the argument or purpose of the work, though data may be presented in a less than completely effective format or some parts of the explication may be uneven. Uses quantitative information, but does not effectively connect it to the argument or purpose of the work. Presents an argument for which quantitative evidence is pertinent, but does not provide adequate explicit numerical support. (May use quasi-quantitative words such as "many," "few," "increasing," "small," and the like in place of actual quantities.)

First Project Question How do the Lumina Degree Qualification Profile (DQP) benchmark statements for Quantitative Fluency align with the more formative VALUE (LEAP) rubric for Quantitative Literacy? Assumption: VALUE rubric will be assessment and curriculum development tool for project faculty use

Our Mapping Results: DQP QF Benchmarks & VALUE (LEAP ) QL rubric Capstone 4 Milestones 32 1 Interpretation Ability to explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words) Provides accurate explanations of information presented in mathematical forms. Makes appropriate inferences based on that information. For example, accurately explains the trend data shown in a graph and makes reasonable predictions regarding what the data suggest about future events. Provides accurate explanations of information presented in mathematical forms. For instance, accurately explains the trend data shown in a graph. Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line. Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means. For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends. Representation Ability to convert relevant information into various mathematical forms (e.g., equations, graphs, diagrams, tables, words) Skillfully converts relevant information into an insightful mathematical portrayal in a way that contributes to a further or deeper understanding. Competently converts relevant information into an appropriate and desired mathematical portrayal. Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate. Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate. CalculationCalculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations are also presented elegantly (clearly, concisely, etc.) Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations attempted are either unsuccessful or represent only a portion of the calculations required to comprehensively solve the problem. Calculations are attempted but are both unsuccessful and are not comprehensive. Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work. Uses the quantitative analysis of data as the basis for competent judgments, drawing reasonable and appropriately qualified conclusions from this work. Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work. Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Explicitly describes assumptions and provides compelling rationale for why each assumption is appropriate. Shows awareness that confidence in final conclusions is limited by the accuracy of the assumptions. Explicitly describes assumptions and provides compelling rationale for why assumptions are appropriate. Explicitly describes assumptions.Attempts to describe assumptions. Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized) Uses quantitative information in connection with the argument or purpose of the work, presents it in an effective format, and explicates it with consistently high quality. Uses quantitative information in connection with the argument or purpose of the work, though data may be presented in a less than completely effective format or some parts of the explication may be uneven. Uses quantitative information, but does not effectively connect it to the argument or purpose of the work. Presents an argument for which quantitative evidence is pertinent, but does not provide adequate explicit numerical support. (May use quasi-quantitative words such as "many," "few," "increasing," "small," and the like in place of actual quantities.) RED CELLS (LEVEL 3) REPRESENT LUMINA DQP ASSOCIATE DEGREE BENCHMARKS. WE ASSUME LEVEL 4 (CAPSTONE) REPRESENTS LUMINA DQP BACHELOR DEGREE BENCHMARKS.

Project Strategies for Success Each faculty member receives a stipend Project coach on each campus Shared (between institutions) Drop Box folder Intercampus planning team includes faculty members Coaches and/or designees offer individual feedback to draft assignments during development Opportunities for discipline-based groups of faculty from both institutions to meet and discuss curriculum Identification of common areas of struggle for students leading to: Scaffolding of curriculum from level courses, Anchored with cumulative assessments that provide contextualization for students to develop and use essential intellectual skills (e.g. QR)

Disciplinary Themes & Assessments BIOLOGYBUSINESSCRIMINAL JUSTICE PSYCHOLOGY GraphingFinancial AnalysisData LiteracyCorrelations Developing graphs Interpreting graphs Calculation Graphing Building tables Analysis, and interpretation Mining for credible data resources Making meaning of data resources Evaluating data resources 100-level: Identifying correlations 200-level: Exploring correlations and Correlation Research 300-level: Testing Correlations Calculations done in Excel and/or SPSS, contextually relevant tools

Assessment Example (Psychology) Design correlational research study to answer correlational research question. Answer should include: Research question Description of Variables Operational definitions of variables (if necessary) Collect data with your group. Each person should collect data from 5 subjects. Using Excel, enter data, create a scatterplot, and calculate Pearson's r correlation coefficient. Determine whether the correlation coefficient is statistically significant at.05 alpha level. Print spreadsheet (data, spreadsheet, scatterplot). Write up your results. Response should include research question, definition of terms and answer to your research question, including the following information: type of correlation (positive, negative, no correlation) and how you know based on the scatterplot and the Pearson r effect size (Pearson r) and what it tells you about the strength of the correlation statistical significance and what it means

Assessment Example (Psychology) Choose two rank ordered or continuous variables within child psychopathology, one must be symptoms of a disorder, the other may be symptoms of another disorder, or it may be a common variable such as a measure of stress, or SES. You should choose a general age range to focus on. Review the literature (at least four articles) on those variables and develop a hypothesis about the relationship you expect to find. You will be provided with a data set in SPSS that contains data on your two main variables. Using SPSS, analyze the correlation between your two variables. Graph the correlation and interpret the correlation based on the graph. Examine the output and interpret the output. How strong is the correlation? What direction is it in? Is it significant? Is it what you hypothesized, based on your reading? Finally, in a page, sum up the findings from the four articles you reviewed, and reflect on what your own data findings mean, in terms of the literature you reviewed. Be sure to reflect on additional variables that you did not measure, might be important to examine in a future study, and speculate about how those variables might influence the correlation you examined.

Assignment Assessment Using the VALUE QL rubric, evaluate the level of student achievement of QL required by this assessment.

Year 2 QC Summit Strengthening our Use of the DQP “The Lumina Degree Qualifications Profile (DQP): Implications for Assessment”, Peter Ewell, 2013 (Afterward, Carol Geary Schneider): 1.pdf 1.pdf “Subdivided assessment strategies” being used by piloting campuses 5 DQP areas (specialized knowledge; broad integrative knowledge; intellectual skills; applied learning; civic learning) intended to “foster integrative and adaptive learning” Subdividing doesn’t promote integrative learning

Year 3 Challenge Integration of DQP Broad Integrative Knowledge standard into assessment development At the associate level, for each of the core areas studied, the student Describes how existing knowledge or practice is advanced, tested and revised. Describes and examines a range of perspectives on key debates and their significance both within the field and in society. Illustrates core concepts of the field while executing analytical, practical or creative tasks. Selects and applies recognized methods of the field in interpreting characteristic discipline-based problems. Assembles evidence relevant to characteristic problems in the field, describes the significance of the evidence, and uses the evidence in analysis of these problems. Describes the ways in which at least two disciplines define, address and interpret the importance of a contemporary challenge or problem in science, the arts, society, human services, economic life or technology. At the bachelor’s level, the student Frames a complex challenge or problem from perspectives and literature of at least two academic fields Proposes a “best approach” to the question or challenge using evidence from those fields. Produces an investigative, creative or practical work that draws on theories, tools, methods from at least two fields. Explains a contemporary or recurring challenge or problem from the perspective of at least two academic fields Explains how the methods of inquiry and/or research in those disciplines can be used to address the challenge Judges the likelihood that combination of disciplinary perspectives and methods contribute to the resolution of the challenge Justifies the importance of the challenge in a social or global context

Year 3 Project Revision 200-LEVEL ASSESSMENT TEMPLATE Focus on one study Students: Describe the study Identify the research question and methodology Describe the results Evaluate the study Articulate some debate in the field or related to the study issue Use the data from the study to communicate (written, oral, multi-media, dramatic) an insightful, creative and/or new perspective on the study issue to authentic audience(s) 400-LEVEL ASSESSMENT TEMPLATE Students explain then analyze the framing of a complex scientific, social, technological, economic or aesthetic challenge from the perspectives and literature of at least 2 academic fields Students propose a “best approach” to the question or challenge using evidence from those fields.

Project Implications & Challenges Discipline-based cross-institutional vertical curriculum design teams Sustainable collaboration – stipends, opportunities for collaboration Change in focus – from inputs (course descriptions) to learning outcomes - for articulation, transfer Faculty investment in assessment results Differences in teaching & learning focus on campuses Departmental dissemination for impact Use of LEAP VALUE rubrics – formatively and summatively

For More Information: Elise Martin, Dean of Assessment, Middlesex Community College, Cathy Pride, Assistant Professor, Psychology, Middlesex Community College, Alice Frye, Lecturer, Psychology, University of Massachusetts, Lowell,