Developing Assessment Methods at Classroom, Unit, and Campus-Wide Levels Presented at The Robert Gordon University Aberdeen, Scotland March 19, 2004 Presented by Trudy W. Banta Vice Chancellor Planning and Institutional Improvement Indiana University-Purdue University Indianapolis 355 N. Lansing St., AO 140 Indianapolis, Indiana iupui.edu

Outcomes Assessment The process of providing credible evidence of the outcomes of higher education undertaken for the purpose of improving programs and services within the institution. Banta, T. W.

Plan Implement Assess Improve

ASSESSMENT... “a rich conversation about student learning informed by data.” -- Ted Marchese -- AAHE

Assessment of Individual Student Development  Assessment of basic skills for use in advising Placement Counseling  Periodic review of performance with detailed feedback  End-of-program certification of competence Licensing exams External examiners

Key Results of Individual Assessment  Faculty can assign grades  Students learn their own strengths and weaknesses  Students become self- assessors

A Second Look  Across students  Across sections  Across courses

 Where is learning satisfactory?  What needs to be retaught?  Which approaches produce the most learning for which students?

Group Assessment Activities Classroom assignments, tests, projects Questionnaires for students, graduates, employers Interviews, focus groups Program completion and placement Awards/recognition for graduates Monitoring of success in graduate school Monitoring of success on the job

Use of Results of Group Assessment Program improvement Institutional and / or state peer review Regional and / or national accreditation

Some Purposes of Assessment 1. Students learn content 2. Students assess own strengths 3. Faculty improve instruction 4. Institutions improve programs/services 5. Institutions demonstrate accountability

Outcomes Assessment Requires Collaboration  In setting expected program outcomes  In developing sequence of learning experiences (curriculum)  In choosing measures  In interpreting assessment findings  In making responsive improvements

Barriers to Collaboration in the Academy 1. Graduate schools prepare specialists 2. Departments hire specialists 3. Much of our scholarship is conducted alone 4. Promotion and tenure favor individual achievements -- interdisciplinary work is harder to evaluate

Campus Interest in Assessment WHAT WORKS in…. H increasing student retention? H general education? H use of technology in instruction? H curriculum in the major?

Good assessment is good research...  An important question  An approach to answer the question  Data collection  Analysis  Report -Gary R. Pike (2000)

Most Faculty Are Not Trained as Teachers FACULTY DEVELOPMENT Can Help Instructors: l Write clear objectives for student learning in courses and curricula l Individualize instruction using a variety of methods and materials l Ask questions that make students active learners l Develop assessment tools that test higher order intellectual skills

Organizing for Assessment

Taxonomy of Educational Objectives (Bloom and Others, 1956) Cognitive domain categories Knowledge Comprehension Application Analysis Synthesis Evaluation Sample verbs for outcomes Identifies, defines, describes Explains, summarizes, classifies Demonstrates, computes, solves Differentiates, diagrams, estimates Creates, formulates, revises Criticizes, compares, concludes

Learning Outcomes for English Majors Graduates can: 1 Demonstrate how language influences intellectual and emotional responses 2 Apply knowledge of rhetorical context by writing effectively and appropriately in context 3 Assess critically spoken, written, and visual representations 4 Apply research strategies appropriate to area of study 5 Synthesize diverse issues and responses raised in collaborative discussions of texts

Learning Outcomes in Science 1. Define and explain basic principles, concepts, theories of science 2. Identify characteristics that distinguish math and science from each other and from other ways of obtaining knowledge 3. Illustrate how developments in science can raise ethical issues 4. Solve theoretical or experimental problems in science 5. Evaluate the validity and limitations of theories and scientific claims in interpreting experimental results 6. Evaluate scientific arguments at a level encountered by informed citizens

In each course Statement Of Competence Teaching/ Learning Strategy Assessment Methods Critical Thinking Group Project Written Paper

Planning for Learning and Assessment

Organizational Levels for Assessment National Regional State Campus College Discipline Classroom Student

Direct Measures of Learning Assignments, exams, projects, papers Indirect Measures Questionnaires, inventories, interviews - Did the course cover these objectives? - How much did your knowledge increase? - Did the teaching method(s) help you learn? - Did the assignments help you learn? GOOD ASSESSMENT INCLUDES BOTH

Fast Feedback (at end of every class)  Most important thing learned  Muddiest point  Helpfulness of advance reading assignments for day’s work in class  Suggestions for improving class / assignments Bateman & Roberts Graduate School of Business University of Chicago

Student Suggestions for Improvement  Install a portable microphone  Increase type size on transparencies  Leave lights on when using projector  Don’t cover assigned reading in detail  Provide more examples in class

Wingspread Seven Principles for Good Practice in Undergraduate Education (Chickering & Gamson, 1987) 1. Student-faculty contact 2. Cooperation among students 3. Active learning 4. Prompt feedback 5. Time on task 6. High expectations 7. Respect for diverse talents and ways of learning

Student Learning Oriented Course Evaluation 1. Learners held high expectations for one another 2. Learners interacted frequently with others 3. Learners participated in learning teams 4. Learners respected diverse talents and ways of learning -Cournoyer Advances in Social Work – Fall 2001

Primary Trait Scoring Assigns scores to attributes (traits) of a task STEPS  Identify traits necessary for success in assignment  Compose scale or rubric giving clear definition to each point  Grade using the rubric

Can Develop a Research Paper 1. Narrows and defines topic 2. Produces bibliography 3. Develops outline 4. Produces first draft 5. Produces final draft 6. Presents oral defense Out- standing Accept- able Unaccept -able

Bibliography Outstanding – References current, appropriately cited, representative, relevant Acceptable – References mostly current, few citation errors, coverage adequate, mostly relevant Unacceptable – No references or containing many errors in citation format, inadequate coverage or irrelevant

Mapping Course Outcomes to Program Outcomes OutcomesCourse 1Course 2Course 3 1  2  3  4  5  6  7 

Sophomore Competence in Mathematics (Multiple choice responses & supporting work) Score Criterion Clear conceptual understanding, consistent notation, logical formulation, complete solution Adequate understanding, careless errors, some logic missing, incomplete solution Inadequate understanding, procedural errors, logical steps missing, poor or no response Problem not attempted or conceptual understanding totally lacking Ball State University

Assessment in Sociology and Anthropology Focus groups of graduating students  Given a scenario appropriate to the discipline, a faculty facilitator asks questions related to outcomes faculty have identified in 3 areas: concepts, theory, methods.  2 faculty observers use 0-3 scale to rate each student on each question  GROUP scores are discussed by all faculty Murphy & Goreham North Dakota State University

Assessment of Group Interaction The Student Participant:  Listened to others  Actively contributed to discussion  Challenged others effectively  Was willing to alter own opinion  Effectively explained concepts/insights  Summarized/proposed solutions 5=Consistently excellent 3=Generally satisfactory 1=Inconsistent and/or inappropriate

Capstone Course in Business Comprehensive case study analysis Graded using scoring rubric covering  writing skills  knowledge of historic and current theories and practice  ability to integrate tools and techniques from business specialties in analyzing the case Olney & Menger St. Mary’s University

In a Comprehensive Assessment Program... INVOLVE t Students t Faculty t Student Affairs Professionals t Administrators t Graduates t Employers

Student Advisory Council at Montevallo A way to provide continuous student assessment Student Recommendations 1 Develop a statement of expected ethical behaviors for students 2 Add a second research course with lab 3 Increase comparative psychology 4 Add terminals for statistics lab 5 Increase opportunities for research, writing, and speaking

Collaboration in Evaluation (Harvard and Samford) H Involve a student team in evaluating a course or program  Interaction increases  Students develop communication skills teamwork competence self-confidence understanding of others’ perspectives  Involvement in learning increases

PRINCIPLES OF UNDERGRADUATE LEARNING (PULs) 1. Core communication and quantitative skills 2. Critical thinking 3. Integration and application of knowledge 4. Intellectual depth, breadth, and adaptiveness 5. Understanding society and culture 6. Values and ethics Approved by IUPUI Faculty Council May 1998

Student Electronic Portfolio  Students take responsibility for demonstrating core skills  Unique individual skills and achievements can be emphasized  Multi-media opportunities extend possibilities  Metacognitive thinking is enhanced through reflection on contents - Sharon J. Hamilton IUPUI

Virginia Commonwealth University 1. First-year English students wrote 1 response / week for 15 weeks 2. Faculty-student affairs teams read essays 3. Sample of writers interviewed 18 months later Responses to findings: Central advising center, new advising handbook, multicultural workshops

Involving Employers Combination of survey and focus groups for employers of business graduates  Identified skills, knowledge, personality attributes sought by employers  Encouraged faculty to make curriculum changes  Motivated students to develop needed skills  Strengthened ties among faculty, students, employers - Kretovics & McCambridge Colorado State University

Colorado State University College of Business Curriculum changes based on employer suggestions:  1 credit added to Business Communications for team training and more presentations  Ethics & social responsibility now discussed in intro courses  New Intro to Business course emphasizing career decision-making  More teamwork, oral & written communication, problem-solving in Management survey courses - Kretovics & McCambridge

Program Course Measurement Findings Action Objective Objective Method Needed Apply CT Use Lab exercises 64% of Students skills in SW statistical ___________ students ask for practice analysis to (80% of scored 3.0 more lab inform students will or better time; 5 days practice earn 3.0 or on lab of lab decisions better on lab exercises added exercises Columbia College

Authentic Assessment at Southern Illinois University - Edwardsville  Business - Case Study Analysis with Memo  Education - Professional Portfolio  Psychology - Poster on Research Project  Engineering - Senior Design Project  Nursing - Plan of Care for Patient

Responses to Assessment at Southern Illinois University-Edwardsville Business – More case studies and research Education – More practice in classroom management Psychology – Curriculum change in statistics Engineering – More practice in writing and speaking Nursing – Simulation lab with computerized patients

Assessment in Fine Arts — Portfolio review and/or audition for every student every semester by Panel of faculty, students, community representatives, staff or faculty outside fine arts Results - Students creativity, conceptualization, technique have improved — Alec Testa — Eastern New Mexico University

Assessment Essentials 1. Plan for assessment 2. Involve all stakeholders 3. Develop goals and objectives 4. Select or design assessment methods 5. Report assessment findings 6. Promote the use of assessment findings 7. Evaluate the assessment program - Palomba & Banta

Success Factors 1 Committed leadership 2 Collaboration between faculty and student affairs leaders 3 Teamwork in planning and implementation 4 Supportive campus climate Concern for students, continuous improvement 5 Involvement in design of assessment 6 Results effectively communicated 7 Conscientious follow-up 8 Persistence

The Future  Need for evidence of accountability will increase  More faculty will recognize benefits of assessment  More electronic assessment methods will be developed  More sharing of assessment methods will take place  Faculty will learn more about learning and student learning will improve