Assessment of Learning in the Liberal Arts Core Donna Vinton, Ph.D. Director of Academic Assessment January 11, 2010

Topics for Today Background on LAC assessment at UNI assessment data for the LAC Questions for next steps

LAC Assessment: Some Background

2000: NCA re-accreditation team recommended improvement in student outcomes assessment processes related to the LAC 2001: LAC Committee formed a subcommittee to develop a comprehensive student outcomes assessment program for the LAC 2002: Pilot testing of Academic Profile and the Collegiate Assessment of Academic Proficiency/Critical Thinking.

2004: The LAC Assessment Subcommittee decided to use Academic Profile for LAC assessment. (This instrument was later renamed Measure of Academic Proficiency and Progress and, more recently, Proficiency Profile.) Initial results of testing with Academic Profile were shared with the President and his cabinet, the Provost and Academic Affairs Council, and the Faculty Senate NCA Progress Report on the General Education Program was submitted to the North Central Association and approved.

2005-6: First year to administer NSSE First use of Oral Communication sections to administer MAPP to first-year students NSSE & MAPP data placed on password-protected page on the Academic Assessment web site Workshops on LAC data offered on campus Presentations on NSSE and MAPP data at department faculty meetings

3 Key Measures of Learning National Survey of Student Engagement (NSSE) This instrument replaced College Student Experience Questionnaire, or CSEQ, named in the original plan. Graduating Senior Survey from UNI Office of Institutional Research Measure of Academic Proficiency and Progress (MAPP, now known as Proficiency Profile and formerly called Academic Profile)

Two Types of Data Student perceptions (indirect measures) –NSSE –Graduating Senior Survey Standardized testing (direct measure) –Measure of Academic Proficiency and Progress

NSSE Survey Content Student Behaviors in College Institutional Actions And Requirements Institutional Actions And Requirements Student Reactions to College Student Background Information Student Background Information Student Learning & Development

Data From NSSE Responses for individual survey items by response stems and item mean Calculated mean for 5 “benchmark” areas: level of academic challenge, active and collaborative learning, enriching educational experiences, student-faculty interaction, supportive campus environment Comparison of UNI data with data from selected cohort groups (e.g., UNI peers, Carnegie peers, Foundations of Excellence institutions) Additional data from consortium questions—American Democracy Project and Consortium for the Study of College Writing

MAPP MAPP, or the Proficiency Profile measures: proficiency in critical thinking, reading, writing and mathematics in the context of humanities, social sciences and natural sciences academic skills developed, versus subject knowledge taught, in general education courses Source: highlights addedhttp://

MAPP: Norm-Referenced Scores Norm-referenced scores (scaled scores) compare the scores of one student or group of students to another. Eight norm-referenced scores: Total Score Skills subscores (critical thinking, reading, writing and mathematics) Context subscores (humanities, social sciences and natural sciences) UNI scores in each of these areas can be compared to scores of students in comparable (Master’s I and II) institutions.

MAPP: Criterion-Based Scores Criterion-referenced scores (proficiency classifications) measure the level of proficiency obtained on a certain skill set. Nine criterion-referenced scores: Mathematics (Level 1, Level 2, Level 3) Writing (Level 1, Level 2, Level 3) Reading (Level 1, Level 2, Critical Thinking) UNI scores in each of these areas can be compared to scores of students in comparable (Master’s I and II) institutions.

Uses of MAPP/NSSE Data at UNI Assessment of learning in the Liberal Arts Core Key performance indicators for the Student Affairs Division %20KPIs% pdf Foundations of Excellence (study of the first-year at UNI) Academic Advising assessment plan Voluntary System of Accountability College Portrait

Assessment Data For the Liberal Arts Core

Tips for Viewing the Data Slides Focus on the types of data available for now; examine specific data later. Think about potential goals for student performance and how we can/should go about setting such goals. Identify areas that are most important—to you, your department, the university. Consider what particular data sets might imply for the LAC and for curricula and performance in academic majors—in terms of courses, assignments, pedagogy, grading, etc. Ask what additional data might be needed and how/with whom to share the data to put it to use most effectively.

Creating an Assessment Plan Determine learning outcomes Describe learning outcomes in terms of specific behaviors Identify assessment strategies/instruments to provide data related to learning outcomes Analyze and use the resulting data Evaluate/revise assessment strategies

The LAC Assessment Plan Assessment of learning in the LAC Categories conducted through the Category Review process Assessment of student learning related to five overall areas of skill represented in the Liberal Arts Core: o Communication o Information o Thinking o Interpersonal o Quantitative

Communication Students should be able to speak, listen, read, write, and view effectively, adapting appropriately to the audience and material at hand.

Speaking Effectively ExcellentGoodAverageFairPoorCannot evaluate Graduating Senior Survey Rate how well UNI has prepared you for speaking effectively (Q1)

Speaking Effectively, continued YearVery MuchQuite a BitSomeVery Little First-Year28%47%22%3% Seniors29%45%20%6% NSSE 11d. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in speaking clearly and effectively?

Listening Effectively ExcellentGoodAverageFairPoorCannot evaluate 38.3%48.7%11.0%1.4%0.4%0.2% Graduating Senior Survey Rate how well UNI has prepared you for listening effectively (Q 4)

Writing Effectively ExcellentGoodAverageFairPoorCannot evaluate 36.1%52.3%9.3%1.7%0.2%0.4% Graduating Senior Survey Rate how well UNI has prepared you for communicating through writing (Q2)

Writing Effectively, continued YearVery MuchQuite a BitSomeVery Little First-Year29%48%18%4% Seniors31%45%21%4% NSSE 11c. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in writing clearly and effectively?

Writing Effectively, continued YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for writing Possible range:

Writing Effectively, continued SkillYearProficientMarginalNot Proficient UNIETSUNIETSUNIETS Writing, Level 1 First-year54%55%31%29%14%15% Seniors67%71%27%21%6%8% Writing, Level 2 First-year13%12%34%33%53%55% Seniors24%22%34%40%42%38% Writing, Level 3 First-year4%5%19%21%77%75% Seniors9% 23%32%68%59% MAPP Proficiency classifications for writing UNI scores compared to scores from ETS testing in Master’s I and II institutions,

Information Students should be able to use both traditional sources and modern technologies to access, analyze, and manage information.

Information ExcellentGoodAverageFairPoorCannot evaluate Understanding written communication. (Q3) 33.0%52.1 %11.9 %1.9 %0.5 %0.6 % Learning new things (Q11) 44.9%44.1%9.7%0.9%0.2% Bringing info/ ideas together from different areas (Q13) 36.8%48.3%12.2%1.9%0.4%0.5% Using research skills (Q14) 37.0%45.6%13.9%2.7%0.5%0.4% Graduating Senior Survey Rate how well UNI has prepared you for:

Information, continued YearVery MuchQuite a BitSomeVery Little First-Year28%47%22%4% Seniors38%41%19%3% NSSE 11g. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in using computing and information technology?

Information, continued YearVery MuchQuite a BitSomeVery Little First-Year29%48%19%4% Seniors27%48%20%5% NSSE 11j. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the area of learning effectively on your own?

Thinking Students should be able to address complex issues and problem situations with sound reasoning, reflective judgment, creative imagination, and a critical, analytical bent of mind.

Thinking ExcellentGoodAverageFairPoorCannot evaluate Defining problems (Q9) 30.6%50.3%16.7%1.6%0.3%0.5% Solving problems (Q10) 33.6%51.2%13.3%1.2%0.3%0.4% Thinking creatively (Q12) 38.3%44.7%13.5%2.4%0.8%0.2% Making decisions (Q19) 39.8%48.5%10.5%0.7%0.3%0.2% Graduating Senior Survey Rate how well UNI has prepared you for:

Thinking, continued YearVery MuchQuite a BitSomeVery Little First-Year37%44%17%2% Seniors45%42%11%2% NSSE 11e. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the area of thinking critically and analytically?

Thinking, continued YearVery MuchQuite a BitSomeVery Little First-Year20%38%35%7% Seniors19%40%31%10% NSSE 11m. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the area of solving complex real-world problems?

Thinking, continued YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for critical thinking Possible range:

Thinking, continued SkillYearProficientMarginalNot Proficient UNIETSUNIETSUNIETS Reading, Level 1 First-year51%52%24%25%26%23% Seniors66%72%20%17%14%11% Reading, Level 2 First-year25%22%17%19%59% Seniors39%41%19%21%42%38% Reading, Level 3 (Critical Thinking) First-year3%2%11%8%86%90% Seniors9%6%15%17%76%77% MAPP Proficiency classifications for writing UNI scores compared to scores from ETS testing in Master’s I and II institutions,

Interpersonal Students should understand human emotions, motivations, and idiosyncrasies, and be able to participate effectively in relationships, groups, and citizenship activities.

Interpersonal: Understand human emotions, motivations, and idiosyncrasies ExcellentGoodAverageFairPoorCannot evaluate Working with people of diverse backgrounds (Q21) 40.4%36.5%16.7%4.1%1.7%0.6% Working as a team (Q22) 44.5%42.8%11.2%0.7% 0.2% Leading others(Q23) 40.6%42.1%14.3%2.0%0.5%0.4% Graduating Senior Survey Rate how well UNI has prepared you for:

Interpersonal, continued: Understand human emotions, motivations, and idiosyncrasies YearVery MuchQuite a BitSomeVery Little First-Year33%42%23%2% Seniors41%43%13%3% NSSE 11h. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in working effectively with others?

Interpersonal: Participate Effectively in Relationships, Groups, and Citizenship Activities ExcellentGoodAverageFairPoorCannot evaluate Planning projects (Q8) 32.9%46.9%15.4%3.4%0.5%0.9% Conducting yourself in a professional manner (Q15) 49.6%40.2%8.3%1.1%0.3%0.5% Upholding ethical standards (Q16) 45.2%41.8%10.8%0.9%0.6% Graduating Senior Survey Rate how well UNI has prepared you for:

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities ExcellentGoodAverageFairPoorCannot evaluate Adapting to change (Q17) 39.4%45.2%13.6%1.1%0.3%0.4% Working under pressure ((Q18) 45.8%43.4%9.3%1.1%0.2% Working independently (Q20) 47.6%42.4%9.0%0.6%0.3%0.2% Graduating Senior Survey Rate how well UNI has prepared you for:

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearVery MuchQuite a BitSomeVery Little First-Year29%30% 11% Seniors16%25%32%28% NSSE 11i. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in voting in local, state, or national elections?

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearVery MuchQuite a BitSomeVery Little First-Year18%27%43%11% Seniors18%33%36%12% NSSE 11l. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in understanding people of other racial and ethnic backgrounds?

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearVery MuchQuite a BitSomeVery Little First-Year24%32%35%9% Seniors22%37%28%13% NSSE 11n. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in developing a personal code of values and ethics?

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearVery MuchQuite a BitSomeVery Little First-Year14%27%38%20% Seniors14%31%39%16% NSSE 11o. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in contributing to the welfare of your community?

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearStrongly Agree DisagreeStrongly Disagree First-Year5%44%43%8% Seniors6%35%46%13% NSSE/American Democracy Project Consortium 16a. I believe my coursework is preparing me to participate in politics (e.g., voting, volunteering or contributing to a campaign, or persuading others about a candidate or political issue).

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearStrongly Agree DisagreeStrongly Disagree First-Year14%53%26%8% Seniors17%53%24%6% NSSE/American Democracy Project Consortium 16b. I believe my coursework is preparing me to discuss politics and government with my friends and family.

Interpersonal, continued: Participate Effectively in Relationships, Groups, and Citizenship Activities YearVery MuchQuite a BitSomeVery Little First-Year33%42%23%2% Seniors41%43%13%3% NSSE 11h. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in working effectively with others?

Quantitative Students should be able to make effective use of quantitative data, and to intelligently apply relevant mathematical and statistical concepts and methods on appropriate occasions.

Quantitative ExcellentGoodAverageFairPoorCannot evaluate 32.4%41.5%18.9%4.7%0.6%1.8% Graduating Senior Survey Rate how well UNI has prepared you for making basic calculations and computations (Q6)

Quantitative, continued YearVery MuchQuite a BitSomeVery Little First-Year28%44%25%3% Seniors30%45%21%4% NSSE 11f. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in analyzing quantitative problems?

Quantitative YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for mathematics Possible range:

Quantitative, continued SkillYearProficientMarginalNot Proficient UNIETSUNIETSUNIETS Mathematics, Level 1 First-year45%47%27%29%27%24% Seniors63%58%27%26%10%16% Mathematics, Level 2 First-year20% 26%27%54%53% Seniors33%30% 29%37%41% Mathematics, Level 3 First-year6%4%11%12%82%84% Seniors13%8%14%18%73%74% MAPP Proficiency classifications for writing UNI scores compared to scores from ETS testing in Master’s I and II institutions,

Humanities YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for humanities Possible range:

Social Sciences YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for social sciences Possible range:

Natural Sciences YearMean Score25 th percentile50 th percentile75 th percentile First-Year Seniors MAPP Norm-referenced subscore for natural sciences Possible range:

Final Thoughts

Evaluating the LAC Strongly Agree DisagreeStrongly Disagree Not Sure Overall quality of most LAC courses is excellent 14.6%58.5%14.5%3.1%9.3% The purposes of most LAC courses are very clear 14.9%52.8%19.5%4.2%8.7% Most of the LAC courses I took at UNI were intellectually demanding. 14.1%55.6%19.0%4.0%7.3% I believe the LAC has been an important part of my education. 18.2%50.9%17.5%5.4%8.1% Graduating Senior Survey Indicate the extent to which you agree or disagree with the following statements.

Evaluating the LAC, continued YearVery MuchQuite a BitSomeVery Little First-Year44%40%14%2% Seniors45%39%14%2% NSSE 11a. To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in acquiring a broad general education?

Evaluating the LAC, continued YearExcellentGoodFairPoor First-Year41%50%9%1% Seniors41%49%8%1% NSSE 13. How would you evaluate your entire educational experience at this institution?

Next Steps: Some Questions How can the data we have be put to use? What additional data/evidence of learning would we like to have, and how can we collect it? How will revision of the LAC affect assessment strategies? Do we want longitudinal data, and if so, how can we test enough students to gather the necessary data? The sample size for seniors is small ( per year for 2007 through 2009). How can we increase it? How do we increase student motivation to complete tests and surveys and do their best on them?

Thank you! Questions/thoughts?