1. Spring 2015 Kyle Stephenson
Clinical Psychology
Spring 2015
Kyle Stephenson
Remember to bring sample reports

2. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing
Last class we talked about interviews, which are a pretty general tool to get at information that a person can actually report to you. Today, we’re going to focus in on two specific targets of assessment – intelligence and personality – that are A) some of the most common targets of assessment, and B) difficult for people to report for themselves (it sounds pretty silly to ask, “how smart are you on a scale from 0-100?”). As with most lectures in this class, this is not a comprehensive lesson in these areas. During graduate school, students typically spend 1-2 semesters learning about each of these assessment areas, and that’s just to start! However, you can choose to specialize in either of these areas during grad school and beyond, so I want to introduce them to you to see if you’re interested enough to take a more in-depth look at the research in these areas.

3. Types of Reliability
Some concepts that are important in assessment are reliability and validity. Most students are not super excited to learn about these concepts and they do sound pretty dry on the surface. But, they are the primary factors that determine whether you’re able to accurately measure any variable in psychology. So, if you’re interested in actually measuring something interesting like intelligence or anxiety or whatever, you need to know a bit about these concepts. Reliability is typically a measure of consistency – see above.

4. Types of Validity
Validity assumes reliability, but goes beyond it. It’s essentially asking the question of whether you’re measuring what you want to measure.
Example – height as a measure of intelligence.
This would probably be a reliable measure (consistent over time, a lot of inter-rater reliability, different measurements would probably correlate highly). But, the measurement would be totally invalid because, obviously, height is not a good indicator of intelligence!

5. Assessment Procedures
Reliability and validity are important when measuring any construct in psychology.
So, why are we spending additional time on IQ and personality assessment? Primarily because they’re some of the most common things assessed by clinical psychologists. We’re talking 85-93% of practicing psychologists using some scale of IQ or personality at least occasionally. Also, these topics tend to be pretty interesting.

6. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing

7. Defining intelligence (or trying to)
Three classes of definitions
Emphasis on adjustment to environment
Adaptability in new situations
Capacity to deal with a range of situations
Emphasis on the ability to learn
How well can you do in school?
Emphasis on abstract thinking
Ability to use symbols and concepts
Verbal and numerical symbols
There have been THOUSANDS of papers, books, etc. that have tried to define intelligence. The difficulty with defining this (and most) factors in psychology is that intelligence is a “construct” – a conceptual idea that represents something abstract that we think will be helpful. In other words, there’s no way to “objectively” measure intelligence because intelligence doesn’t actually exist in any physical form. Instead, we use the idea of “intelligence” as a stand-in for a web of related skills and abilities that allow us to talk about and measure things we think are important.
Three important outcomes that we think depend on intelligence are mentioned above. As a society, we generally think that a “smart” person should be able to adjust to their environment, learn things, and think abstractly. Certainly, other societies might place more or less emphasis on any on of these abilities and, correspondingly, the way they define intelligence would probably also change.
An important take-home point here is that there’s no objectively correct answer as to what intelligence is – we just have to decide what we want intelligence to be. But, we can determine which definitions are most helpful and, once we’ve defined intelligence and selected the things it should be related to, we can objectively judge better or worse measures of it.

8. Theories of Intelligence
This graphic is a good overview of the most common type of theory of intelligence – a hierarchical one. This type of theory holds that people have some overall level of intelligence (g, for general intelligence), but that there are multiple components of intelligence that can vary significantly between people who have the same “overall” intelligence. For example, Jim and Jennifer could both have the same IQ, but Jim might have higher levels of spatial intelligence while Jennifer may have high levels of number intelligence.

9. Theories of Intelligence
Gardner’s multiple intelligences
Problem solving skills
8 formal groupings
linguistic
musical
logical-mathematical
spatial
bodily-kinesthetic
naturalistic
interpersonal
intrapersonal
Major criticism: “Intelligences” vs. “talents”
But, there are other types of theories. One of the most famous is Gardner’s theory of “multiple intelligences.” This theory basically does away with the concept of overall intelligence, and also expands the range of skills that are included in “intelligence.” In addition to the major criticism of whether we should consider, for example, musical ability as intelligence, this theory also doesn’t do very well at explaining why many different skills sets are highly correlated. For example, linguistic intelligence tends to be correlated with spatial and logical indicators of intelligence, which could potentially be accounted for by an overarching level of “overall intelligence.”
You do NOT need to memorize this theory (won’t be tested on it) – I simply include it as an example of a way of defining intelligence that’s not well reflected on modern IQ tests.

10. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing

11. Clinical Assessment Wechsler Scales Deviation IQ based
Originally designed for adults
Performance scale and
Verbal scale
Each made up of subtests
Each subtest gets an IQ score
Full Scale IQ
But we can’t wait until we have a perfect definition. There are many reasons that we need to assess intelligence now, including assessment of learning disabilities (impairments in one area of ability like reading or math that is far outside what would be expected given a person’s general level of intelligence), diagnosis of mental retardation and other developmental disorders, and identification of particularly skilled individuals (helpful for contexts like the army, schools, etc.).
By far the most common way to assess intelligence is one of the Wechsler scales, which include the WAIS, WISC, and WIPSI. All of these scales work in generally the same way.
IQ (intelligence quotient) is thought to be normally distributed in the population, meaning that most people fall in the same small range (85-115), with few falling outside that range. Importantly, because we know the exact shape of a normal distribution, we can know based on your score what percentage of the population falls above and below you on any given scale. This method is called “deviation based,” because your score and percentile is based on how much your score deviates from the average performance.

12. Aptitude vs. Achievement
It’s important to note that the Wechsler scales are, for the most part, meant as measures of aptitude. Aptitude is essentially your potential or ability to learn and do things. In contrast, achievement is what you’ve already done with that potential.
For grad school, there are measures of each – a general exam that tries to measure aptitude, and a psychology-specific exam that tries to measure the knowledge you’ve accumulated about the field.
Content knowledge vs. potential to attain knowledge

13. Clinical Assessment Wechsler Adult Intelligence Scales (WAIS-IV)
Ages 16-90
Scoring system
Index scores: Verbal Comprehension, Perceptual Reasoning, Working Memory, Processing Speed
Full Scale IQ Score
15 subtests
Reversal items
Subtests are meant to get at different aspects of intelligence, and to make it possible to identify strengths and weaknesses.
Reversal items save time and maintain rapport with test takers.

14. How does the examination look?
Generally verbal questioning
Stimulus booklet
Block design
While almost all the responses are verbal, there has been a concerted effort over the past 20 years to include more non-verbal measures. One big reason for this is that verbal questions and content are more likely to be biased against people from backgrounds different from the people constructing the test (generally well-educated wealthy white people who live in the northeast). For example, asking about supply and demand or even something as concrete as parts of a boat may not do a good job of assessing intelligence for a Latina single mother who has lived in Arizona her whole life (never been on a boat) and didn’t graduate high school. It’s not that she’s not intelligent, she just hasn’t had the life experiences to be equipped to answer those questions.
You don’t need to memorize the parts of the WAIS – I include this information just to give you a sense of how the test looks.

15. WAIS-IV Picture Completion Subtest
Block design and picture completion are two examples of attempts to have less culturally-biased measures in the test.

16. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing
The WAIS and other scales generally do a good job of reliably measuring IQ. But, there are a lot of other issues when it comes to intelligence. What about validity – are these measures related to what they should be? Are they useful in predicting behaviors and performance? What do they tell us about the construct of intelligence?

17. Age and IQ Stability
First, we generally expect intelligence to be stable and it looks like IQ scores are generally stable, especially once you become an adult.

18. IQ
We would also expect IQ to be heritable to some degree, but not 100%. This table highlights the importance of both genes and the environment.

19. Here are some of the environmental factors that are associated with lower IQ scores.
There are numerous factors that predict IQ scores. For example, most ethnic minority groups in the US tend to score lower on these tests. Men tend to score slightly higher on spatial ability and women tend to score slightly higher on verbal ability. (see your text for more detail)
Does this mean that some of these groups are “smarter” than others? No. There could be numerous reasons for these differences. For one, ethnic minority families on average are exposed to more of the risk factors listed above than caucasian families. Much of the difference in IQ scores can be accounted for by these differences in contextual and cultural factors. Remember, just because two things are correlated (like ethnicity and IQ) doesn’t mean that one causes the other (e.g., minority ethnicity causes lower IQ scores). There can be a third variable (like stress, quality of schooling, etc.) that explains why the original two variables are related. This does NOT mean the test is biased (see below for more discussion about test bias).

20. IQ predicts
Academic achievement (GPA & number of school years completed)
Job performance
Income (and child’s income)
SES
Type of occupation
Contact with justice system
In many ways, the most important question is not whether we are “really” measuring intelligence or what intelligence “really” is, but instead whether our measures are useful. Do they give us information that allows us to make better decisions? The answer is generally yes. You can have a better prediction of someone’s future behavior based on knowing their IQ score. This doesn’t mean that IQ necessarily causes these outcomes, but it does allow us to prepare for them and even intervene.
E.g., identify children at risk of struggling academically and provide additional support, select job applicant who will do the best, etc.
.4 to .6 between IQ and GPA
.3 to .5 – job performance based on supervisor ratings
Parent IQ scores predict 1/6 of variance in their children’s income
Within-family studies of IQ-SES link. If a person has a higher IQ than family average, he/she is more likely to achieve higher SES and vise versa.
-.19 correlation to juvenile offenses (but may be that lower IQ just more likely to get caught)

21. But… Test motivation / incentives
Incentives can increase IQ scores by 9.6 IQ points
When taking motivation (assessed by objective observers) into account, predictive ability of IQ scores decreases significantly, especially for non- academic outcomes.
But, as with all complicated questions, there are some caveats here. One big one is motivation – how invested are you in doing well on this IQ test? The answer can impact your score (which it shouldn’t – intelligence shouldn’t depend on how motivated you are today), which calls into question how much the association between IQ and other outcomes is a result of differences in motivation to succeed (rather than raw ability or potential to succeed).
Comes back to the point that it’s often easier to determine whether a test if “useful” (predicts important outcomes) than to determine whether it’s “valid” (actually measures “intelligence”).

22. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing

23. What do we mean by “personality?”
Enduring traits (not states/moods)
Should be somewhat consistent across time and situations
Should predict behaviors
Needs to be specific, measurable, and meaningful
Not, “he’s a good guy”
Rather, “he is X amount of Y trait, which means he is more/less likely to Z”

24. Objective vs. Projective
“Objective” is misleading
Really just more objective than projective tests
Application, scoring, and interpretation are more objective
“Objective” really refers to the fact that different psychologists giving the same test would get the same result. But, most measures of personality rely on the person’s subjective report (e.g., “how often do you try new things?”).

25. Objective Testing Advantages Economical Large group testing
Computer scoring & interpretation
Single dimension or trait can be targeted
Straightforward
Objective & reliable
Economical – you can give the client the test and walk away. You can have multiple people (e.g., child and parent) complete tests at the same time.
Scoring – can score test very quickly, essentially like a scantron, or even use a computer to administer and score almost instantly.
Single dimension – you don’t need to know the person’s entire history and background to understand their responses
Objective/reliable – should be minimally biased by clinician, and should give similar results across time and situations

26. Objective Testing Disadvantages
Questions may be uncharacteristic of respondent
Underlying reason for behavior not evident
Unrepresentative behaviors
Distorted or lost information
Misinterpretation
Uncharacteristic – e.g., if question asks about relationship with siblings and client doesn’t have any siblings – what does their response actually measure?
Underlying – e.g., if question asks about frequency of fighting, you don’t know whether it’s because client has to protect themselves from aggressive peers, whether he/she likes boxing, or whether he/she is aggressive and dangerous.
Unrepresentative – e.g., asking questions about sleep habits during finals week
Biases – people often want to misrepresent themselves, making themselves seem either worse off or better of than they are. E.g., faking good in context of assessment for work, faking bad in context of disability claim.

27. Objective Testing Empirical criterion keying
Minnesota Multiphastic Personality Inventory (MMPI)
No assumption based on content of item
What matters is that those in the diagnostic group share responses
Empirical basis, not always rational
One way to help address misrepresentation is using empirical criterion keying. This method is different from how tests are usually constructed:
Make large pool of potential items based on some mixture of expert opinion and information directly from population of interest
Pilot items and remove confusing ones, ones that overlap with one another too much, etc.
Re-test new version and establish structure (e.g., subscales), reliability, and validity
Makes additional changes based on results
Re-test and publish
An important aspect of this typical process is that the initial criterion for inclusion of items is usually face validity – the questions appear to measure what they are in fact measuring. If we want to know how often you feel sad, we ask you how often you feel sad. This is usually fine, but can cause problems when misrepresentation is a concern.
Empirical criterion keying is DIFFERENT in that the surface-level content of the items doesn’t matter. If people who say they prefer Washington to Lincoln are consistently more likely to develop schizophrenia, it can be a valid predictor whether or not it makes any sense. Makes it hard to fake bad or fake good, but also can make it more difficult to interpret results.

28. Objective Testing
Here’s a nice summary of different ways of establishing the validity of an item in an objective test.

29. Simulated MMPI Items
MMPI is widely used, especially in research but also fairly commonly in treatment.
Again, don’t memorize this, just included b/c I thought you’d be interested.

30. Objective Testing
The NEO is another very commonly used measure and is based on a theory suggesting there are 5 primary aspects of personality.

31. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing

32. Projective Testing Characteristics Examinee imposes own structure
Unstructured stimulus
Indirect methodology
Freedom of response
Many variables to rate
Projective testing was very popular in the 50’s-70’s, less so now.
On the upside, it allows you to assess aspects of the person’s experience that he/she may not be comfortable talking about directly or may not even be aware of. You can see the strong link between psychoanalytic theory and these methods of assessment. But, even other clinicians will use them in forensic settings, and with children.

33. Projective Testing Rorschach inkblots Multiple variation in scoring
10 cards
Administration
Card is shown, responses offered & noted
Evaluation and rating of answers

34. Rorschach Inkblot
Evaluation of responses in complicated, takes a long time, and is often not supported by research. Responses rated on a number of continua including how much of the blot is used, whether movement is implied, whether color is taken into account, etc.

35. Overview – Day 6 Reliability & validity Intelligence assessment
Defining intelligence
Assessment methods
Notes about IQ
Personality assessment
“Objective” testing
Projective testing
Use & abuse of testing

36. Use & Abuse of Testing Protection Privacy
APA standards require training
Restrictions in purchasing tests
Privacy
Right to explanations
Relevancy of testing
Informed consent
Test results can have a large impact on people’s lives so it’s important that trained professionals administer, score, interpret, and report results. It’s also important in many cases that the people taking the tests have not seen them beforehand (if you were able to practice the items on the WAIS beforehand it would cease to be a valid measure of IQ). So, you need credentials to purchase many of the tests and the psychological community all has a stake in keeping the content of some tests from the general public.
Test takers also have important rights including explanations as to what the tests measure and how, that the information provided by the test is relevant to their goals, and that they consent to testing. In some cases tests are court ordered, but the more common situation is that tests are administered for research or therapy purposes, in which case participants/clients need to be fully informed.

37. Use & Abuse of Testing Test Bias
Validity issue: if it can be demonstrated that the validity of a test varies significantly across groups, then a case can be made that the test is “biased”
Things to remember:
Differences in mean scores do not necessarily indicate test bias
Tests may be valid (and not biased) for some purposes, but not for others
One can “overcome” test bias by using different prediction equations for the different groups
We’ve already touched on test bias a bit and it’s an important concept to return to.
One of the most common examples is the mean difference in IQ scores between different ethnic groups in the U.S. It’s important to realize that the existence of such a difference doesn’t in and of itself mean the test is biased (remember other factors determine IQ scores besides intelligence like motivation, SES, educational background, etc.). A test is biased if it is NOT AS VALID for different groups. For example, if IQ scores did a good job of predicting college GPA for Caucasian, but not African American students. This type of bias was certainly present in the past, but has in many ways been eliminated or accounted for in more modern versions. For example, the best tests now use entire samples of a single ethnic group (e.g., Native American) and establish reliability and validity within that group, which can result in different scoring methods being used for different people. Again, if one group scores worse on a measure and that score is still equally as valid, it is not test bias. A clear example is that most ethnic minorities tend to do worse on IQ tests, and also tend to have a harder time in school. In this case, the test is doing what it’s supposed to do (predicting future academic performance) and is thus not “biased” as that term is used in psychological science.

38. Take-Home
Psychological assessment is often not measuring physical “things” in the world, but rather theoretical constructs
However, we can still objectively test the reliability and validity of assessments and judge them based on their usefulness
Intelligence and personality testing are the most common assessments performed by psychologists and each comes with its own set of controversies and complexities