1. Foundational Perspectives on Student Development: Implications for Accountability and Assessment
George D. Kuh
Council on University
Planning and Analysis
Guelph, Ontario
June 23, 2009

2. Context Global Competitiveness in Degree Attainment
The New Majority and Demographic Gaps
Questionable Levels of Student Performance 2

3. Context Global Competitiveness in Degree Attainment
The New Majority and Demographic Gaps
Questionable Levels of Student Performance
In an Environment of Increasing Fiscal Strain…
 We Need Higher Levels of Student Achievement 3

4. Inquiring Minds Want to Know…
What frameworks and perspectives about student learning, development, and success can be instructive in guiding assessment of essential learning outcomes?
How do they fit in with the current press on institutional accountability?

5. Overview The context for accountability and assessment
Theoretical and empirical perspectives on student development and success
Implications

6. Accountability 2009
Greater emphasis on student learning outcomes and evidence that student performance measures up
Demands for comparative measures
Assessment “technology” has made great strides, but is insufficient to document learning outcomes most institutions claim
Increased calls for transparency ---public disclosure of student and institutional performance

7. Two Paradigms of Assessment
Continuous Improvement
Accountability
Strategic dimensions
Purpose
Formative (improvement)
Summative (judgment)
Stance
Internal
External
Predominant ethos
Engagement
Compliance
Implementation
Instrumentation
Multiple/triangulation
Standardized
Nature of evidence
Quantitative and qualitative
Quantitative
Reference points
Over time, comparative, established goal
Comparative or fixed standard
Communication of results
Multiple internal channels
Public communication, media
Use of results
Multiple feedback loops
Reporting
Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting
for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco. 7

8. Purposes of Assessment
Evidence -- Demonstrate value and return on investment for accountability purposes
Improvement – Determine what is working and what is not to improve programs, courses, departmental outcomes.
Transparency -- Provide information about educational effectiveness and what students know and can do.
Feedback -- Enable students to see their progress and learn how to improve.

9. What’s Happening
Over past 5 years institutions report…
More assessment activities; greater attention to accreditation requirements
Emphasis on clearer statements of student learning outcomes, but little student understanding of outcomes
More emphasis on engaged learning practices including learning communities, service-learning courses, internships, senior capstones, and undergraduate research
Little transparency, even less documented improvement
(Ewell, 2008; Maki, 2004; Trends in Learning Outcomes, General Education, and Assessment, AAC&U, 2009)

10. Accountability Triangle
Assessment
Improvement
Transparency

11. Student Development
Outcomes: Host of desirable skills, knowledge, beliefs and attitudes, persistence, educational attainment
Process: Unfolding of human potential toward increasingly complicated, refined functioning

12. Knowledge acquisition & application Humanitarianism
Outcome Domains
Cognitive complexity
Knowledge acquisition & application
Humanitarianism
Interpersonal and intrapersonal competence
Practical competence
Source: Kuh, 1993

13. Cognitive complexity
Cognitive skills including reflective thought, critical thinking (e.g., ability to summarize information accurately and perceiving logical coherences and discernable themes and patterns across different sources of information), quantitative reasoning, and intellectual flexibility (i.e., openness to new ideas and different points of view).

14. Knowledge acquisition & application
Understanding knowledge from a range of disciplines and physical, geographic, economic, political, religious, and cultural realities, and the ability to relate knowledge to daily life including using information presented in one class in other classes or other areas of life.

15. Humanitarianism
Understanding and appreciation of human differences, including an increased sensitivity to the needs of others and contributions to common good.

16. Interpersonal & intrapersonal competence
A coherent, integrated constellation of personal attributes (e.g., identity, self-esteem, confidence, integrity, appreciation for the aesthetic and spiritual qualities of life and the natural world, sense of civic responsibility) and skills (e.g., how to work with people different from oneself).

17. Practical competence
Skills reflecting an enhanced capacity to manage one’s personal affairs (e.g., time management, decision making), to be economically self-sufficient, and to be vocationally competent.

18. Association of American Colleges and Universities18

19. Narrow Learning is Not Enough: The Essential Learning Outcomes
Knowledge of Human Cultures and the Physical & Natural World
Intellectual and Practical Skills
Personal and Social Responsibility
“Deep” Integrative Learning 19

20. Deep, Integrative Learning
Attend to the underlying meaning of information as well as content
Integrate and synthesize different ideas, sources of information
Discern patterns in evidence or phenomena
Apply knowledge in different situations
View issues from multiple perspectives

21. Theoretical Perspectives: Student Development
Psychosocial (Chickering, 1968; Chickering & Reisser, 1993; Erikson, 1968; Cross, 1995)
Cognitive-structural (Perry, 1970; Kitchner & King, 1981, 1990; Baxter-Magolda, 1992; 2001; Kohlberg, 1969, 1984; Gilligan, 1977, 1986)
Typology (Cross, 1976; Kolb, 1976, 1984; Holland, 1968, 1997; Kuh et al., 2000)
Person-environment interaction (Astin, 1968, 1993; Barker, 1968; Horowitz, 1987; Pace & Stern, 1958, Moos, 1979

22. Theoretical Perspectives: Persistence & Educational Attainment
Sociological (Astin, 1977, 1993; Tinto, 1975, 1987, 1993; Marsden, 2004; Pascarella & Terenzini, 1991, 2005)
Organizational (Bean, 1983; Berger & Braxton, 1998)
Psychological (Bandura, 1982; Bean & Eaton, 2000; Dweck, 2000)
Cultural (Bourdieu & Passeron, 1977; Kuh & Love, 2000; Rendon et al, 2000; Tierney, 1992)
Economic (Becker, 1964; Braxton, 2003)

23. Pascarella’s (1985) causal model: Effects of college student development

24. Pascarella’s (1985) causal model: Effects of college student development
Students
Aptitude
Achievement
Personality
Aspiration
Ethnicity

25. Pascarella’s (1985) causal model
Institution
Enrollment
Faculty-student ratio
Selectivity
% Residential
Student
development
Students
Aptitude
Achievement
Personality
Aspiration
Ethnicity

26. Pascarella’s (1985) causal model
Institution
Enrollment
Faculty-student ratio
Selectivity
% Residential
Interactions with
faculty
peers
Student development
Students
Aptitude
Achievement
Personality
Aspiration
Ethnicity

27. Pascarella’s (1985) causal model
Institution
Enrollment
Faculty-student ratio
Selectivity
% Residential
Interactions with
faculty
peers
Student development
Students
Aptitude
Achievement
Personality
Aspiration
Ethnicity
Institutional Environment
Quality of student effort
Engagement

28. Major Syntheses of Postsecondary Impact Studies
Jacobs (1956)
Sanford (1962)
Feldman & Newcomb (1969)
Astin (1977)
Bowen (1977)
Pace (1979)
Pascarella & Terenzini (1991)
Astin (1993)
Pascarella & Terenzini (2005)

29. What Matters to Student Success
Kuh, G.D., Kinzie, J., Buckley, J.A., Bridges, B.K., & Hayek, J.C. (2007). Piecing together the student success puzzle: Research, propositions, and recommendations. ASHE Higher Education Report, 32(5). San Francisco: Jossey-Bass.
Commissioned papers at:

30. Pre-college experiences
• Study Habits
• Peer Involvement
• Interaction with Faculty
• Time on Task
• Motivation
• Other
• First Year Experience
• Academic Support
• Campus Environment
• Time on Task
• Peer Support
• Teaching & Learning
Approaches
• Other
Pre-college experiences

31. Pre-college Characteristics Associated with Student Success
Academic preparation

32. Pre-college Characteristics Associated with Student Success
Academic preparation
Ability and college-level skills
Financial wherewithal
Family education and support

33. Early College Indicators of Persistence and Success
Goal realization
Psycho-social fit
Credit hours completed
Academic and social support
Involvement in the “right” kinds of activities

34. What Really Matters in University: Student Engagement
Because individual effort and involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement.
Pascarella & Terenzini, 2005, p. 602

35. Foundations of Student Engagement
Time on task (Tyler, 1930s)
Quality of effort (Pace, s)
Student involvement (Astin, 1984)
Social, academic integration (Tinto,1987, 1993)
Good practices in undergraduate education (Chickering & Gamson, 1987)
College impact (Pascarella, 1985)
Student engagement (Kuh, 1991, 2005)
Pace (1970s)
Pioneer in this movement of looking at the entire student experience versus just looking at test scores or grades to assess student learning.
He asked questions related to students’ academic and social experiences in college – assessed how much effort they were putting forth in their educational experiences
Astin (Hired by Pact at UCLA in 1980s)
Promoted his theory of student involvement
Amount of learning taking place directly proportional to quantity and quality of energy invested in educational activities
Tinto (Also in the 1980s)
Retention model – greater social and academic integration, both formal and informal processes -> greater satisfaction -> more likely to stay
Chickering and Gamson (1980s analysis of hundreds of studies over several decades)
1) Student-faculty contact, 2) Cooperation among students, 3) Active learning, 4) Prompt feedback, 5) Time on task, 6) High expectations, and 7) Respect for diverse talents and ways of learning
Kuh (1990s – idea of student engagement)
What students do -- time and energy devoted to educationally purposeful activities
What institutions do -- using effective educational practices to induce students to do the right things

36. Student Engagement Propositions
What students do -- time and energy devoted to educationally purposeful activities
What institutions do -- using effective educational practices to induce students to do the right things
Educationally effective institutions channel student energy toward the right activities

37. Good Practices in Undergraduate Education (Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)
Student-faculty contact
Active learning
Prompt feedback
Time on task
High expectations
Respect for diverse learning styles
Cooperation among students

38. National Survey of Student Engagement (pronounced “nessie”) Community College Survey of Student Engagement (pronounced “cessie”)
College student surveys that assess the extent to which students engage in educational practices associated with high levels of learning and development

39. Student Engagement Initiatives
2,500,000+ students from 1,400 different schools
80+% of 4-yr U.S. undergrad FTE
50 states, Puerto Rico
64 Canadian IHEs
Australasian Survey of Student Engagement (AUSSE)
South African Survey of Student Engagement (SASSE)

40. Grades, persistence, student satisfaction, and engagement go hand in hand

41. It’s more complicated than this…
Many of the effects of college are “conditional”
Some are compensatory

42. NSSE: Who’s more engaged?
Women
Full-time students
Students who live on campus
Students with diversity experiences
Students who start and stay at the same school
Students in learning communities

46. Next Steps
Develop additional indicators of success for different types of students.

47. Additional Indicators
course retention
transfer student success
success in subsequent courses
degree/certificate completion
graduate school
employment
capacity for lifelong learning
focus on gaps in success
Bullet 1: DEEP Schools are confident enough of who they are and what they are striving for that they are willing to question whether they are performing as well as they can. This sense of institutional efficacy contributes to a collaborative culture of improvement.
Bullet 2: DEEP schools are never quite satisfied with their level of performance. They continually revisit and rework policies and practices to make them marginally better.
Bullet 3: Confidence combined with a focus on improved performance makes DEEP schools dynamic organizations. Because they are not afraid to experiment and invest in ideas, they are inclined toward innovation. Even those that are cash strapped typically find small amounts of money to fuel the flame of innovation. 47

48. High Impact Activities
First-Year Seminars and Experiences 
Common Intellectual Experiences
Learning Communities
Writing-Intensive Courses
Collaborative Assignments and Projects
“Science as Science Is Done”; Undergraduate Research
Diversity/Global Learning
Service Learning, Community-Based Learning
Internships
Capstone Courses and Projects 48

49. Essential Learning Outcome: NSSE Deep/Integrative Learning
Integrating ideas or information from various sources
Included diverse perspectives in class discussions/writing
Put together ideas from different courses
Discussed ideas with faculty members outside of class
Discussed ideas with others outside of class
Analyzing the basic elements of an idea, experience, or theory
Synthesizing & organizing ideas, info., or experiences
Making judgments about the value of information
Applying theories to practical problems or in new situations
Examined the strengths and weaknesses of your own views
Tried to better understand someone else's views
Learned something that changed how you understand an issue 49

50. Effects of Participating in High-Impact Activities
on Deep/Integrative Learning and Gains

51. Effects of Participating in High-Impact Activities
on Student Engagement

52. High Impact Activities Increase Odds Students Will:
Invest time and effort
Interact with faculty and peers about substantive matters
Experience diversity
Get more frequent feedback
Reflect & integrate learning
Discover relevance of learning through real-world applications
Bullet 1: DEEP Schools are confident enough of who they are and what they are striving for that they are willing to question whether they are performing as well as they can. This sense of institutional efficacy contributes to a collaborative culture of improvement.
Bullet 2: DEEP schools are never quite satisfied with their level of performance. They continually revisit and rework policies and practices to make them marginally better.
Bullet 3: Confidence combined with a focus on improved performance makes DEEP schools dynamic organizations. Because they are not afraid to experiment and invest in ideas, they are inclined toward innovation. Even those that are cash strapped typically find small amounts of money to fuel the flame of innovation.

54. High-Impact Practices and the Disparities Within…
Fewer 1st gen students
Fewer students of color
Fewer transfer students
Fewer part-time students
Fewer older students
Bullet 1: DEEP Schools are confident enough of who they are and what they are striving for that they are willing to question whether they are performing as well as they can. This sense of institutional efficacy contributes to a collaborative culture of improvement.
Bullet 2: DEEP schools are never quite satisfied with their level of performance. They continually revisit and rework policies and practices to make them marginally better.
Bullet 3: Confidence combined with a focus on improved performance makes DEEP schools dynamic organizations. Because they are not afraid to experiment and invest in ideas, they are inclined toward innovation. Even those that are cash strapped typically find small amounts of money to fuel the flame of innovation.

55. High-Impact Practices
Assessing Student Engagement in
High-Impact Practices
To what extent does your institution provide these experiences?
[√ = have on campus; √ = required; estimate the % of various student populations in these activities]
Learning Community
First Year Seminars
Research w/ Faculty
On Our Campus
Required for all
% Students involved
% First Generation
% Transfer Students
% African American
% Latino Students
% Asian American
% other
% Adult Students

56. 56

57. Next Steps
Develop additional indicators of success for different types of students.
Verify the institutional policies and practices that work best with different groups of students at different types of institutions.
Further develop, validate, and refine assessment tools.

58. Sample Data Sources Locally-developed measures National instruments
National Survey of Student Engagement (NSSE)
Beginning College Survey of Student Engagement (BCSSE)
Faculty Survey of Student Engagement (FSSE)
Cooperative Institutional Research Program (CIRP)
Your First College Year (YFCY)
College Student Experiences Questionnaire (CSEQ)
Noel Levitz Student Satisfaction Inventory
ETS MAPP and Major Field Tests
ACT Collegiate Assessment of Academic Proficiency
Collegiate Learning Assessment (CLA)
Institutional data -- GPA, financial aid, transcripts, retention, certification tests, alumni surveys, satisfaction surveys…
Electronic portfolios
Exercise: What kind of data do you have or could you have that could be linked to college experience data to better understand and serve the campus community?

59. Next Steps
Determine appropriate ways to measure, report, and use student success indicators for accountability and improvement (e.g., common template)

60. Student engagement varies more within than between institutions.

61. Variation With-in
Quality is not uniform within institutions. The lion’s share of the variation is among students, within institutions

62. Next Steps
Determine appropriate ways to measure, report, and use student success indicators for accountability and improvement (e.g., common template)
Determine what postsecondary institutions can realistically do at what cost to help academically underprepared students overcome the deficiencies they bring with them to college.

63. Let’s Do Four Things
1. Measure things that matter.
We value what we measure, so wise decisions are needed about what to measure in the context of campus mission, values, and desired outcomes.

64. Evidence of College Graduates Skills/Knowledge
Very effective
Fairly effective
Supervised internship/community-based project
83%
Senior project (e.g., thesis, project)
79%
Essay tests
60%
Electronic portfolio & faculty assessments
56%
Multiple-choice tests
32% 64

65. Focus on mission-relevant, educationally meaningful indicators linked to student success that the institution can do something about
Bullet 1: DEEP Schools are confident enough of who they are and what they are striving for that they are willing to question whether they are performing as well as they can. This sense of institutional efficacy contributes to a collaborative culture of improvement.
Bullet 2: DEEP schools are never quite satisfied with their level of performance. They continually revisit and rework policies and practices to make them marginally better.
Bullet 3: Confidence combined with a focus on improved performance makes DEEP schools dynamic organizations. Because they are not afraid to experiment and invest in ideas, they are inclined toward innovation. Even those that are cash strapped typically find small amounts of money to fuel the flame of innovation.

66. Let’s Do Four Things
1. Measure things that matter.
2. Use theory and research to inform assessment efforts
3. Adopt and adapt best practices in assessment, public reporting, and improvement
4. Resolve the perceived tensions between improvement and accountability

67. Two Paradigms of Assessment
Continuous Improvement
Accountability
Strategic dimensions
Purpose
Formative (improvement)
Summative (judgment)
Stance
Internal
External
Predominant ethos
Engagement
Compliance
Implementation
Instrumentation
Multiple/triangulation
Standardized
Nature of evidence
Quantitative and qualitative
Quantitative
Reference points
Over time, comparative, established goal
Comparative or fixed standard
Communication of results
Multiple internal channels
Public communication, media
Use of results
Multiple feedback loops
Reporting
Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting
for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco. 67

68. What’s Next for Student Learning Outcomes Assessment?
Multiple measures
Diagnostic, milestone, and culminating assessments all necessary
Wide range of assessment practices – basic to more complex assessments of learning environment, student work, direct learning outcomes, “value-added”
Measures of college-level learning by different majors, fields, levels
Demonstrate action on assessment results
Quality improvement – measuring change, real improvements in student learning

69. AAC&U VALUE Rubrics Inquiry and analysis Critical thinking
Creative thinking
Written communication
Oral communication
Quantitative literacy
Information literacy
Teamwork
Problem solving
Civic knowledge and engagement
Intercultural knowledge and competence
Ethical reasoning and action
Foundations and skills for lifelong learning
Integrative learning

70. AAC&U VALUE project – 12 Rubrics

71. Documenting what students learn and can do is of growing interest both on campus and with its many constituencies.
Know far too little about what actually happens in assessment on campuses around the country.
NILOA is conducting a survey and other activities to help fill that void.
NILOA’s mission is to identify best practices in assessment that institutions find useful.

72. Questions
&
Discussion