1. Lessons Learned from Industry: Achieving Diversity & Efficacy in College Success ETS - College Board Invitational Conference Washington, DC Wayne Camara & Krista Mattern September 8, 2008

2. 2 Job Analysis Organizations use job analysis to determine what work outcomes are desired. Sample individual outcomes (productivity, job performance, retention) and organizational outcomes (efficiency, quality, innovation, team work) Identify performance components (pc) pc = {Declarative knowledge x Procedural knowledge x Motivation} Knowledgex Cognitive skills x Level of effort Goals Interpersonal “ Persistence “ Ability, Interests, Education, Experience Importance & Prob. Of Outcomes Predictors

3. 3 Job Analysis – College Success 1.Identify desired performance outcomes for individuals and organizations (college success) (GPA, return, graduate, life after college – grad school, certification) Each outcome likely has different predictors 2.Identify performance tasks associated with outcomes (persistence, academic ability, health, engagement) 3.Identify or develop performance measures (GPA, advisory ratings, self report data, dB of student engagement)

4. 4 Predicting Performance Performance Tasks (GPA, Engagement, Graduation) Competencies (Knowledge, Skills, Abilities) Predictors (Tests, Essay, Ref, Application) Goldstein, Zedeck, & Schneider (1993)

5. 5 Group differences are not unique to tests: They are present across most educational measures

6. 6 Large Mean Differences Persist on Cognitive Ability Tests by Race/Ethnicity Remain SAT CR+M

7. 7 Source: U.S. Census Bureau College-Going Rates of High School Graduates Aged 18 to 24 by Ethnic Group, 1999-2006

8. 8

9. 9 Disparities Exist in HS Graduation, HS Drop Out and College Ready Source: Manhattan Institute, Public HS Graduation and College-Readiness Rates: 1991-2002, http://Manhattan-institute.org/html/ewp_08.htm;* Condition of Education, 2007 Table 23-2 http://Manhattan-institute.org/html/ewp_08.htm

10. 10 Graduation Rates in 2004 by ethnicity Published 3/7/2007 Title Awards conferred by Title IV institutions, by race/ethnicity, level of award, and gender: United States, academic year 2004–05 (recalculated to eliminate students who with other or no ethnicity reported). http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf; Public HS graduation rates: WICHE 3/2008, http://www.wiche.edu/policy/knocking/1992-2022/index.aspAwards conferred by Title IV institutions, by race/ethnicity, level of award, and gender: United States, academic year 2004–05http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf

11. 11 Rationale for looking beyond Grades and Tests What is college success? Is it more than grades and GPA? (Camara & Kimmel) Develop measures that predict your goal or desired outcome. Employers test multiple measures: openness, conscientiousness, extraversion, agreeableness, neuroticism Military use today (GED). Can do does not equal will do.

12. 12 Predictors of College Success Tests Measure Colleges Collect in some form (applications, transcripts) Not Collected in Standardized form College Skills Content Knowledge AchievementNon-Cognitive Personal Qualities/ Experiences/ Characteristics School Performance/ Context Interests - Vocations Verbal ReasoningMathMotivationLettersGradesCareer Interests Math ReasoningLanguage ArtsFollow-throughEssayGPA Learning -Study Skills WritingScienceCommunicationCommunity ServiceWeighted GPAInterest in Major Metacognition Social Studies/ HumanitiesConscientiousnessExtra-curricularRankSelf Efficacy CreativityForeign LanguageLeadershipWork ExperienceCourses CompletedAspirations/ Practical KnowledgeLanguage ProficiencyOther PersonalityLiteracy in Second LangAcademic Rigor Realistic Self- concept Spatial Relations Health/LifestyleTeacher RatingsAP/Honors Courses Intellectual Curiosity AdaptabilityGenderSchool Size Technology – Research Skills EthnicitySchool Quality Ability to Benefit Family Education/ Income Ability to Pay Residence

13. 13 Identify a broader domain of college student performance: Review university mission statements and department objectives Interview with university staff responsible for student life Review of the education literature on student outcomes Our systematic search (A JOB ANALYSIS OF UNDERGRADUATE STUDENTS) resulted in 12 dimensions of student performance… Validate items with successful juniors – they are the experts. Research collaboration with Michigan State University

14. 14 12 Dimensions of Student Performance Broadening the Performance Domain in the Prediction of Academic Success (Schmitt, Oswald, & Gillespie, 2004)  Knowledge, learning, mastery of general principles  Continuous learning, intellectual interest and curiosity  Artistic and cultural appreciation  Multicultural appreciation  Leadership  Interpersonal skills  Social responsibility, citizenship and involvement  Physical and psychological health  Career orientation  Adaptability and life skills  Perseverance  Ethics and integrity

15. 15 Two “Noncognitive” Measures Situational judgment inventory A situation is presented along with several alternative courses of action. The respondent is asked to indicate what she/he would be most likely and least likely to do. Biodata Short, multiple choice reports of past experience/background and interests/preferences.

16. 16 Sample SJI Item for Leadership You are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything.  -1 Look at them until someone eventually says something  Start the conversation yourself by introducing yourself  +1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone  Try to start working on the project by asking everyone’s opinion about the nature of the project  You would take the leadership role by assigning people to do things or ask questions to get things rolling

17. 17 Sample Biodata Items for Leadership  The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was:  4 or more  3  2  1  I did not take a leadership role  How often do you talk your friends into doing what you want to do during the evening?  most of the time  sometimes (about half the time)  occasionally (about as often as others in my group  seldom or infrequently  never

18. 18 Study 1: Develop and refine the measures 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year. Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing measures of personality in predicting college GPA. The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students’ self ratings, and peer- ratings of performance were examined (.19,.22, and.14, respectively).

19. 19 Study 2 Examine Validity & Subgroup Differences: 10 Participating Institutions & 2,700 Freshmen HBCUN Winston-Salem (public)229 Spelman College (private)254 Big Ten (public)N University of Iowa335 Michigan State University546 Ohio State University 304 University of Michigan297 Indiana University170 Other InstitutionsN University of Chicago (private)168 Cal State – Fullerton (public)223 Virginia Tech (public)237

20. 20 Predicting FYGPA: Total Sample across 10 Institutions (N = 2443) Non cognitive measures contribute little beyond tests and grades in predicting academic outcomes

21. 21 Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899 ) However, non cognitive measures will predict non cognitive outcomes – better than tests or grades (graduation, attendance, leadership, engagement)

22. 22 Percent of Students Selected: Two Composites and Three Selection Strategies Top 85% Top 50% Top 15% Group AB AB+ AB AB+ AB AB+ Hispanic 4.4  4.6 4.1  4.9 3.9  5.5 (+.2) (+.8) (+1.6) Asian 7.6  7.7 9.9  9.5 17.5  12.9 (+.1) (-.4) (-4.6) African-American 17.9  19.8 9.6  13.6 1.3  7.2 (+1.9) (+4.0) (+5.9) White 70.2  67.9 76.4  71.9 77.2  74.4 (-2.3) (-4.5) (-2.8) AB = equally weighted composite of HSGPA and SAT/ACT. AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI. Less selective Moderately selective Very selective

23. 23 Correlations of Non-cognitive Measures with Cumulative GPA and Graduation VariableCumulative GPAGraduation SAT/ACT scores 0.590.24 HSGPA 0.560.28 Knowledge 0.240.12 Continuous Learning 0.110.03 Artistic Appreciation 0.220.13 Multicultural Appreciation 0.120.11 Leadership 0.100.14 Responsibility 0.120.15 Health 0.130.08 Career Orientation -0.180.02 Adaptability 0.010.07 Perseverance 0.020.10 Ethics 0.150.11 SJI 0.200.14 Note. Bold values are significant at p<.01. N ranges from 1560 to 1798 across variables. Graduation is dichotomously scored (1, 0).

24. 24 Study 3: Purpose & Research Questions 15 institutions (n = 4,164 for SJI and 7,645 for biodata) Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible Administer new measures to college applicants rather than college freshmen. On an annual basis, collect class absenteeism, self rated performance of the noncognitve dimensions, and commitment to the university from enrolled students; institutions will provide course grades and retention. University of Washington Meredith College Michigan State University University of Southern California Lafayette College Furman University Earlham College University of North Carolina, Chapel Hill Ohio State University Kenyon College Purdue University Gonzaga University Spelman College University of Puget Sound Johnson and Wales University

25. 25 Incremental Validity of Biodata Measures OutcomesN R 2 (HSGPA,SAT)Overall R 2 R2R2 BARS570.023 0.443* 0.420* OCB570.0170.392 0.374* Deviance570.0250.3730.348 Turnover Intent580.0770.2480.172 Academic Satisfaction580.0080.3530.345 Social Satisfaction580.0770.2940.218 FYGPA84 0.201* 0.335*0.134 Absenteeism 580.0610.2340.173 To preserve N in these regressions, the SJI was not included because of a relatively low response rate to this measure. It is worth noting that small sample sizes, such as those observed in these analyses, can seriously limit the ability to detect significant relationships due to decreased statistical power.

26. 26 Next Steps In need of a demonstration project – Implement with Research across a few colleges! Encourage applicants to complete on-line as part of admissions and only use data as a “plus factor.” Provide incentives for applicants to complete the new measures and institutions to track student success over time. Likely outcomes will be more diversity, broader talent, greater retention, and standardized – defensible measures to evaluate applicants fairly and objectively. Increased efficiency and judgmental decisions based on data and comparability. For more information, go to http://www/iopsych.msu/cbstudy