1. The Measurement of Resilience
DO NOT DISSEMINATE
Meeting of the 21st Century Skills Collaboration
18 July 2017
The Measurement of Resilience
David Klieger, J.D., Ph.D.
Research Scientist
Educational Testing Service
Chelsea Ezzo
Jennifer L. Bochenek
Chen Li & Nimmi Devasia
Frederick A. Cline

2. Social Belongingness Theory
Reduce feelings of insecurity in how connected people feel to others (Baumeister & Leary, 1995; Walton & Cohen, 2011)
Increase fit, increase resilience (and/or decrease adversity and thus the need for as much resilience)
National Science Foundation (NSF) and National Academy of Sciences (NAS) (2017) have found sense of belonging promising for predicting persistence and success outcomes, esp. for minority groups
6 of 7 studies with stat. sig. effects (p≤.05) for college GPA ( SDs)
Meta-analysis by Lazowski & Hulleman (2015) showed statistically significant effect of interventions to increase social belongingness ( 𝑑 =.35 SDs; k=5 studies; 95% CI=[.07,.63])
Resilience measures that can predict belongingness/resilience or identify the need to increase belongingness/resilience (due to lack of sufficient belongingness/resilience) potentially can provide a great deal of benefit.

3. The “Diversity-Validity Dilemma” & “High Stakes Selection”
Extremely challenging to maximize both racial-ethnic diversity and predictive validity at the same time (at least in a U.S. context) (Pyburn, Ployhart, & Kravitz, 2008)
However, diversity and predictive validity need not be mutually exclusive goals
Tradeoffs between diversity and predictive validity have been observed for a long time in the United States
Can an assessment of Resilience help decision-makers increase racial-ethnic diversity and predictive validity simultaneously?

4. Selecting a Measure of Resilience
Need to understand what we mean when we say that we are measuring “resilience” (i.e. its construct definition, its structure)
How much and what kinds of adversity contexts are needed for it to be “resilience”?
Should resilience be measured at a single time point, or should it be measured over more than one?
Measure to predict it or measure it for intervention (to build it up in people)?
How do you avoid/minimize risk of respondents’ faking responses/managing impressions to look good/get selected?
Usual psychometric concerns (reliability, validity)

5. Selecting a Measure of Resilience
Examples:
“Lower” stakes – SuccessNavigator® - Likert self-report to improved student retention and graduation
Higher stakes
WorkFORCE Assessment for Job Fit® – forced-choice paired comparisons for selection
“flexibility and resilience” – “adjusting well to changing or ambiguous work environments, handling stress, accepting criticism and feedback from others, being positive even when facing setbacks”
ETS® Personal Potential Index (PPI) – third-party reporting; standardized letter of recommendation with 1-5 Likert scales and text-based comments; for selection

6. SuccessNavigator®

7. ETS’S SuccessNavigator®
Academic Skills
Organization
I make a schedule for getting my school work done.
I take due dates seriously.
Meeting Class Expectations
I attend almost all of my classes.
I complete the reading that is assigned to me.
Self-Management
Sensitivity to Stress
I get stressed out easily when things don't go my way.
I am easily frustrated.
Determination
When feeling stressed about the amount of homework I have, I try to get organized to get on top of my homework.
When worried about a test, I get prepared regardless.
Academic Self-Efficacy
I'm confident that I will succeed in my courses this semester.
I can do well in college if I apply myself.
Test Anxiety
When taking a test, I think about what happens if I don't do well.
Before a test, my stomach gets upset.
Social Support
Connectedness
I feel connected to my peers.
People understand me.
Institutional Support
If I don't understand something in class, I ask the instructor for help.
I know how to find out what's expected of me in classes.
Barriers to Success
Family pressures make it hard for me to commit to school.
People close to me support me going to college.
Commitment
Commitment to College
One of my life goals is to graduate college.
The benefit of a college education outweighs the cost.
Institutional Commitment
This is the right school for me.
I’m proud to say I attend this school.
rxx=.89
(Markle et al., 2013)
rxx=.89
rxx=.89

8. SuccessNavigator® Feedback
Academic Success Index — student's projected GPA range in their first year
Retention Success Index — student's probability of returning in the following year
Course Acceleration Recommendation — recommendations for Math and English acceleration for students at the upper end of cut score bands
General Skill and Subskill Scores — individual scores for the four general skills and 10 subskills
Recommended Next Steps — customized to each student's needs with specific action the student should take to ensure they persist to graduation

9. ETS’s SuccessNavigator®

10. FACETS®

11. FACETS

12. ETS’s FACETS® rxxs in .80s and .90s (Naemi et al., 2014)
Outcome: Overall (Graduate) GPA
Predictors: df
Adjusted R2
UGPA + GMAT
2, 261
.22
UGPA + GMAT + FACETS unit weighted composite
3, 260
.26
Plot of the predicted vs observed scores. The diagonal indicates the case of perfect prediction.
Underpredicted: Points above the diagonal indicate students with higher observed scores than predicted
Overpredicted: Points below the diagonal show students with lower observed scores than predicted.
Facets
Very low
Low
High
Very high
Underpredicted 28 29 34 43
Overpredicted 38 37 32 23
(Kyllonen et al., 2017)

13. FACETS® Other Findings
FACETS scores were strongly uniquely predictive of non-grade outcomes (e.g., contribution to the community, participation in student government) that are related to belongingness
(Kyllonen et al., 2017)

14. ETS PPI®

15. ETS® PPI: 6 Dimensions + An Overall Evaluation Item = 25 Items
1. Knowledge and Creativity
Has a broad perspective on the field
Is among the brightest persons I know
Produces novel ideas
Is intensely curious about the field
2. Communication Skills
Speaks in a clear, organized and logical manner
Writes with precision and style
Speaks in a way that is interesting
Organizes writing well
3. Teamwork
Supports the efforts of others
Behaves in an open and friendly manner
Works well in group settings
Gives criticism/feedback to others in a helpful way
4. Resilience
Accepts feedback without getting defensive
Works well under stress
Can overcome challenges and setbacks
Works extremely hard
5. Planning and Organization
Sets realistic goals
Organizes work and time effectively
Meets deadlines
Makes plans and sticks to them
6. Ethics and Integrity
Is among the most honest persons I know
Maintains high ethical standards
Is worthy of trust from others
Demonstrates sincerity
+ An Overall Evaluation (1-Item)

16. PPI Evaluation Report Applicant information Recipient Evaluator
Ratings for each
dimension and for the overall
evaluation
I first knew Pat when she was a freshman. She was a good student then, but she has really blossomed in the past two years as she has become more confident and experienced. Her honors thesis was excellent – thoroughly researched and well-written. She would do well in any graduate program.
This is the first page of a five-page report. There is information about the applicant on the top. There is also information about the recipient.
Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., SCORELINK and GRE are registered trademarks of Educational Testing Service (ETS)

17. PPI Evaluation Report Each evaluator’s rating for Resilience
Any comments provided by evaluators for the dimension
This is the page that shows Teamwork and Resilience. In this example, one evaluator hasn’t provided any comments for the Resilience dimension. There are no ratings on this dimension from that evaluator, which means he answered “No opportunity to evaluate” for each of the four statements in the Resilience dimension.
Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING., SCORELINK and GRE are registered trademarks of Educational Testing Service (ETS)

18. PPI: Assessing Resilience Can Enhance Diversity: Comparisons of Minority Applicants to White Applicants
How many times more often is the higher-scoring group going to be chosen? (Sackett & Wilk, 1994):
Cohen's d
Admissions Context
0.20
0.40
0.70
1.00
Highly Selective
1.4 times
2.2 times
4.2 times
7.7 times
Selective
1.2 times
2.1 times
3.1 times
Relatively Non-Selective
No Diff.
1.1 times
1.3 times
1.5 times
(Klieger et al., 2014, 2017)

19. % of selected candidates belonging to underrepresented minority groups
PPI: Assessment of Resilience Can Lead to Increase in Selection of Underrepresented Minorities
18%
% of selected candidates belonging to underrepresented minority groups
14% 9% 6%
How much the selection system depends on cognitive assessment versus how much it depends on assessment of resilience
Adapted from Klieger et al., 2014

20. PPI: Internal Consistency Was High Enough to Allow Strong Prediction of Success
(Klieger et al., 2014, 2017)

21. Raters Were Consistent Enough With Each Other to Allow Strong Prediction of Success
Note. Generalizability (g) Theory coefficients are reported here; k = number of graduate institutions (different universities); figures are based on 3 PPI evaluators; models accounted for graduate institution and discipline sector (STEM or non-STEM); the random effects model allows institutions to vary beyond the six from which data were collected.
(Klieger et al., 2014, 2017)

22. PPI: By Third Semester, Resilience Predicts Success Well Compared to Cognitive Skills and Prior Grades
(Klieger et al., 2014, 2017)
Note. k = 6, the number of institutions (different universities); “Aggregate” = sample-size weighted average across STEM and non-STEM. Non-STEM includes mainly coursework in the Social Sciences, Humanities, and Education fields. The subscript (1 or 3) indicates whether a column of data pertains to first or third semester cumulative grade point average.

23. PPI Helps Us Predict Success Even When We Already Have Prior GPA and Standardized (Cognitive) Test Scores to Use
Incremental Validity of PPI Over Prior GPA + Standardized (Cognitive) Test Score for Predicting Future Cumulative GPA as of Semester 1 and Semester 3
Note. k = 6, the number of institutions. “6 Average” = a calculated unit-weighted average of mean ratings for the six main PPI dimensions. “Overall” = mean rating for the one-item Overall PPI dimension. ΔR2 = incremental validity over U.S. UGPA for predicting cumulative graduate GPA. U.S. GPA excludes grades received from institutions outside of the United States. The subscript (1 or 3) indicates whether a column of data represents first or third semester.
(Klieger et al., 2014, 2017)

24. 609-734-1977 (U.S.) dklieger@ets.org
Thank You. Questions?
(U.S.)