Evaluating Writing for Quantitative Reasoning Integrative Learning Project 2005 Summer Institute Carleton College: Scott Bierman, Liz Ciner, Jackie Lauer-Glebov, Carol Rutz, Mary Savina s:

Carleton’s ILP Project ► Cross-cutting literacies and skills ► Tie in with faculty “workload” – improving faculty members’ knowledge of what everyone else is doing. Today: One piece – Connecting quantitative literacy and writing

Questions for you ► How do you evaluate students’ writing on your campus? ► How do you evaluate students’ quantitative reasoning (or literacy or... ) on your campus? ► How might you combine the two?

Outline ► Background: Quantitative Inquiry, Reasoning, and Knowledge (QUIRK) initiative ► Background: Writing Portfolio initiative ► Writing about QUIRK: First-year seminars ► Developing criteria and reading student writing ► Intersecting the cross-cutting literacies: what the future may hold at Carleton

What is QuIRK? ► Inquiry ► Knowledge ► Reasoning ► rk> rk> rk>

Background: Writing Portfolio Initiative ► May 2001 – Faculty vote to institute writing portfolio requirement. Volunteers from class of 2004 submit portfolios in May ► May 2003 – All sophomores in the class of 2005 submit writing portfolios. ► May 2004 – All sophomores in the class of 2006 submit writing portfolios.

The portfolio/QUIRK cycle Carleton College 2004, FIPSE proposal

Background: History of QuIRK ► present– faculty meet informally to systematically discuss concerns about students’ quantitative literacy. ► May 2003 – Background white paper presented to Dean and President ► September 2004 – FIPSE grant approved. ► September 2004 and continuing – First-year seminars offered, course development funds for faculty, speakers, QuIRK faculty workshops ► June QUANT squad formed, portfolio reading and rubric revised ► August 2005 – workshop featuring College of San Mateo learning community – Tools for Thought (Jean Mach and Mike Burke) ► December 2005 – Writing workshop “Writing with Numbers”, John Bean, facilitator ► May 2006 – First writing examples from QuIRK first-year seminars expected in writing portfolios

How can we help students appreciate and strengthen QR? ► First-year seminars ► QuIRK across the curriculum, examples:  Improve QR in Biology lab reports.  Develop a data project on the trans-Atlantic slave trade for use in a History course.  Improve a QR assignment in a Writing Course.  Create a new Political Science course emphasizing comparative electoral analyses. ► Faculty development and campus events

First-year seminars ► IDSC 100, Measured Thinking: Principles of Quantitative Reasoning ► POSC 100, Media and Electoral Politics ► SOAN 100, Myths of Crime ► ENTS 100, Geology and Human Health Other QR assignments in first-year classes, e.g. English 109 (Rutz, Shuffleton)

Evaluation of First-Year Seminars (thanks to Jackie Lauer-Glebov, Carleton Office of Institutional Research) ► 38 students in QR seminars and 45 control students in other FYS seminars did not differ at the pretest on seven QR-related questions on first- week survey (e.g. “I have the skills to read and understand a statistical analysis of data.”).. ► Compared to pretest scores, the posttest responses to the QR questions by both QR and control students were more positive. ► The QR seminar students had more positive responses to the QR questions than the control students.

Evaluating QuIRK in student writing – May/June 2005 ► Norming sessions for “Quant Squad™” ► Quant Squad™ reads 281 portfolios (of about 480 total), flagging 381 essays with some QuIRK content from 102 courses, in 25 departments and programs. ► QuIRK taskforce revises criteria, articulates goals, attends record-setting Minnesota Twins game.

What we saw in student writing ► Students are aware of the power of quantitative claims and quantitative reasoning. ► Many student do not put numbers in context – “is this number a big one?” ► Many students are less specific than they should be and they overuse “most,” “many,” “few,” “seldom.” ► Many students had problems interpreting numbers and using them to advance arguments.

Some faculty development implications ► Articulate the importance of quantitative reasoning to all faculty (choice of workshop topics). ► Rewrite assignments to encourage students to report and interpret the quantitative content of their sources. ► Demonstrate what we mean by “good interpretation” of data (e.g. for gall fly papers). ► Highlight excellent assignments in the arts and humanities that encourage students to use and interpret numbers.

June 2005 goals articulation ► Goals:  Thinks quantitatively  Implements “quantitative analysis” competently  Interprets and evaluates thoughtfully  Communicates effectively

Revised criteria ► States questions under consideration in numerical/quantitative/measurable terms; ► Identifies appropriate quantitative/numerical/empirical evidence to address questions and issues; ► Generates, collects, or accesses appropriate data; ► Investigates questions and issues by selecting and carrying out appropriate quantitative or numerical methods; ► Uses quantitative methods correctly; ► Presents and/or reports the quantitative data appropriately; ► Focuses analysis appropriately on relevant data; ► Interprets results to address questions and issues under consideration; ► Assesses the limitations of the methods employed, if appropriate to the task or assignment.

What’s next at Carleton? – Using student writing to evaluate... ► Visual literacy? (Working group formed January 2005 Visual representations of evidence should be governed by the principles of reasoning about quantitative evidence. For information displays, design reasoning must correspond to scientific reasoning. Clear and precise seeing becomes as one with clear and precise thinking – Edward Tufte (1997), Visual Explanations ► Information literacy? (Mellon pilot project ) ► Group work? (ECC review in progress) ► Integrative learning in individual courses? (HHMI/CISMI) ► Ethical inquiry/reflection and civic engagement? ► ????

Thanks to... ► Neil Lutsky, Sam Patterson, Jackie Lauer-Glebov and the others who wrote the FIPSE grant proposal ► Carol Rutz ► The QUANT Squad ► Corrine Taylor, Director of the Quantitative Reasoning Program at Wellesley College ► Liz Ciner, John Ramsay and the others in the Dean of the College office ► Lynn Steen, Randy Richardson and others outside Carleton who’ve helped us