Evaluating Academic Challenge Beyond the NSSE Jim Purcell, Associate Vice Chancellor Oklahoma State Regents for Higher Education

Legacy of Assessment Letter from the Six Nations to William & Mary College

We are convinc'd, therefore, that you mean to do us Good by your Proposal; and we thank you heartily. But you, who are wise, must know that different Nations have different Conceptions of things; and you will therefore not take it amiss, if our Ideas of this kind of Education happen not to be the same with yours. We have had some Experience of it. Several of our Young People were formerly brought up at the Colleges of the Northern Provinces; they were instructed in all your Sciences; but, when they came back to us, they were bad Runners, ignorant of every means of living in the Woods, unable to bear either Cold or Hunger, knew neither how to build a Cabin, take a Deer, or kill an Enemy, spoke our Language imperfectly, were therefore neither fit for Hunters, Warriors, nor Counselors; they were totally good for nothing.

We are, however, not the less oblig'd by your kind Offer, tho' we decline accepting it; and, to show our grateful Sense of it, if the Gentlemen of Virginia will send us a Dozen of their Sons, we will take care of their Education; instruct them in all we know, and make Men of them.

Knowing is half the Battle GI Joe Cartoon Character and Action Figure Know Thyself Socrates “Every new adjustment is a crisis in self- esteem.”

Assessment is a Mirror Assessment is a reflection of reality – not reality Now we see but a poor reflection as in a mirror. Now I know in part; then I shall know fully... 1 Corinthians 13:12

JoHari Window Known by SelfUnknown by Self Known by Others Unknown by Others Open Area Hidden Area Unknown Area Blind Area Joseph Luft and Harry Ingham

JoHari Window Known by SelfUnknown by Self Known by Others Unknown by Others Hidden Area Unknown Area Blind Area Joseph Luft and Harry Ingham Open Area

Georgia College & State University

Student Engagement Notional value: We like the idea, but do not take action. Cost for caring about student engagement: Investment of time, study, money, and risk. What are we doing? Is it working? Important to take risk: Pause to think day.

The NSSE first Reading Asked the questions that were at the core of the institutional mission Sense of Place Sense of Activity Sense of Interaction Provoked faculty interest Enabled institutional conversation

1= never2= sometimes3= often4= very often Asked questions in class or contributed to class discussions. Selected NSSE findings Seniors st -year From NSSE st -year 3.16 Seniors st -year 3.30 Seniors National Liberal Arts Georgia College

1= never2= sometimes3= often4= very often Worked with other students on projects during class. Selected NSSE findings Seniors st -year From NSSE st -year 2.64 Seniors st -year 2.43 Seniors National Liberal Arts Georgia College

1= never2= sometimes3= often4= very often Worked with other students on assignments outside of class. Selected NSSE findings Seniors st -year From NSSE st -year 2.83 Seniors st -year 2.76 Seniors National Liberal Arts Georgia College

Faculty Response? Bogus: The survey was not administered correctly Bad Survey: The survey failed to measure student engagement Bad Students: We deal with a different type of student than other schools Bubba’s problem: The results do not apply to me and my colleagues in my department

A few months later...

 Used to communicate with an instructor or other students. (Sig..018)  A&S (2.80), Bus (2.90), Ed (3.40), HS (3.40)  Made a class presentation. (Sig..015)  A&S (2.87), Bus (2.69), Ed (3.42), HS (3.07)  Worked with other students on projects during class. (Sig..001)  A&S (2.33), Bus (2.55), Ed (3.32), HS (2.60) National Survey of Student Engagement Senior students –by GCSU Academic Schools Statistically significant responses at.05 level or higher

 Participated in a community-based project as part of a regular class course. (Sig..000)  A&S (1.47), Bus (1.25), Ed (2.21), HS (2.53)  Used an electronic medium to complete or discuss a project. (Sig..044)  A&S (2.27), Bus (2.17), Ed (2.90), HS (2.53)  Discussed grades/assignment with instructor. (Sig..014)  A&S (3.13), Bus (2.52), Ed (3.00), HS (2.73) National Survey of Student Engagement Senior students –by GCSU Academic Schools Statistically significant responses at.05 level or higher

 Number of assigned textbooks, books, or book length packs of course reading. (Sig..005)  A&S (3.24), Bus (2.90), Ed (2.79), HS (3.80)  Number of papers over 20 pages. (Sig..004)  A&S (1.50), Bus (1.50), Ed (1.25), HS (2.00)  Number of papers under 20 pages. (Sig..022)  A&S (2.72), Bus (2.62), Ed (3.40), HS (3.20) National Survey of Student Engagement Senior students –by GCSU Academic Schools Statistically significant responses at.05 level or higher

 Course work emphasizes synthesizing and organizing ideas. (Sig..005)  A&S (2.83), Bus (2.28), Ed (2.70), HS (3.27)  Course work emphasizes making judgments about the value of information. (Sig..027)  A&S (2.79), Bus (2.31), Ed (2.55), HS (3.20)  Preparing for class (studying, reading, etc) (Sig..017)  A&S (3.77), Bus (2.86), Ed (2.95), HS (4.29) National Survey of Student Engagement Senior students –by GCSU Academic Schools Statistically significant responses at.05 level or higher

 Contributed to acquiring a work related skills. (Sig..007)  A&S (3.13), Bus (2.83), Ed (3.45), HS (3.73)  Contributed to analyzing quantitative problems. (Sig.033)  A&S (2.69), Bus (3.00), Ed (2.40), HS (3.20)  Contributed to the welfare of your community. (Sig..005)  A&S (2.07), Bus (2.21), Ed (2.65), HS (3.07) National Survey of Student Engagement Senior students –by GCSU Academic Schools Statistically significant responses at.05 level or higher

A few more months later...

Level of Academic Challenge Active and Collaborative Learning Student Interaction with Faculty Enriching Educational Experience Supportive Campus Environment NSSE survey

Evaluating Academic Challenge Beyond the NSSE Stephen L. Payne, Karynne L. M. Kleine, Jim Purcell, and Ginger Rudeseal Carter page/0,11855, ,00.html page/0,11855, ,00.html Kuh (2001) strongly encouraged institutional leaders to adopt multi-faceted, multi-year, data-collection strategies that capitalize on what contrasting assessment approaches do best.

Eleven items are incorporated in the academic challenge dimension of the NSSE. Many items relate strongly to objectives on Bloom’s and other learning taxonomies and others involve the amount of time or effort that students must have on the course. Academic Challenge

Do students at particular institutions discount the importance of aspects of academic challenge that the NSSE tries to measure? Is the NSSE missing aspects of academic challenge that faculty and students at particular institutions believe would be helpful for improving academic challenge there? How might an institution with a specific mission, academic culture, and set of characteristics examine its degree of academic challenge and move toward more comprehensive assessment and improvement of this academic challenge? Focus Group Questions

Focus Groups Twenty-nine faculty Fifty-four students Main and branch campus students were interviewed. Transcriber Asked questions concerning: conceptions of academic challenge, the appropriateness of the academic challenge items on the NSSE, and what might be done to improve academic challenge at this institution.

Academic Challenge Questions

What is Academic challenge? Initial reactions to the concept of academic challenge were frequently negative – Emphasized the extra time and effort as well as the initial uncertainty or confusion associated with such courses Students later mentioned rewards of academically challenging courses. Synthesis and, application of learning content Faculty teaching style and student interactions were a concern Students

Off-campus students Stressed stronger concern for career entry and success Reacted less favorably than other students to academic challenge in courses, particular in core courses of those connected to their majors

 Significant minority of faculty emphasized lack of student preparation to enable student engagement.  “Basics” limiting student opportunities for higher-order learning and academic challenge  Need for strategies to challenge students.  Need improved resources for faculty development What is Academic challenge? Faculty

AA cademic challenge requires a recognition of different student learning styles and offering multiple paths LL earning outcomes become more important NN eed more and improved assessment using the NSSE and other indicators What is Academic challenge? Faculty

Proposal for expanded assessment Accepted by the Academic Governance Committee of the University Senate of GC&SU (2004) Ask to devote time to developing this internal assessment instrument Used NSSE measures for benchmarks for quality enhancement plan

Critical thinking, reading, and writing are at the core of academic challenge, but these important concerns are not handled the existing NSSE items Several academic challenge items stress student effort Items are not linked to learning outcomes Ideas for local research related to the NSSE discussions

Study relationships between student perceptions of learning and actual outcome measures of affective, creative, aesthetic, and leadership development particular programs Development of items that permit distinction between experiences of courses Distinguish between academic challenge and experiences that are merely highly competitive, demanding, or even intimidating.

 Students have less interest and motivation for learning in core courses they perceive as having less direct career relevance.  What roles of responsibilities might faculty and students assume for improved learning outcomes?  Establishing both high academic rigor in their courses as well as a high level of faculty-student relationships out of class seems to be a contradiction to many faculty. What other interesting tidbits from this study?

Students placed greater responsibility on faculty in their learning experience. Faculty feel that assignments can result in more meaningful learning outcomes than students might realize at this point in their lives What other interesting tidbits from this study?

Closing Remarks We should design educational experiences that are empowering Measure quality of encounters Ensure encounters are transformational We should desire positive restlessness