Appraisal and Its Application to Counseling COUN 550 Saint Joseph College For Week # 4 Copyright © 2004 by R. Halstead. All rights reserved.

Class Topics Validity Administering Tests & Communicating Results Scoring & Interpreting Tests Norms Standard Scores

I was just looking at the tree across the street. The distance variance of the fallen leaves from the trunk of the tree looks like they are normally distributed. Seeing that phenomenon is like looking into the eyes of Dog!!

Test Validity Validity - a means of expressing the degree to which a certain inference drawn from a test is appropriate and meaningful. Stated more simply, validity helps us answer the following question: Does a test measure what it purports to measure?

The Multidimensional Nature of Validity Validity - a means of expressing the degree to which a certain inference drawn from a test is appropriate and meaningful. There are two different types of inferences drawn from tests: Descriptive inferences and Predictive inferences Because validity, in a sense, is multidimensional in nature, one must use a variety of methods to establish if a test is valid.

Categories of Validity Most test manuals refer to three categories of validity Content Validity Criterion Validity Construct Validity

Categories of Validity Content Validity - The items of a test represent that which they are supposed to measure. Content Validity is often supported by expert judgment. Face Validity is also a term used in some test manuals to express that a panel of judges has held that test appears to measure what it purports to measure.

Categories of Validity Criterion-Related Validity - A test score is related to outcome criteria. There are two elements addressed when considering the Criterion Validity Concurrent Validity - the test score agrees (concurs) with other valid measures of the same construct. Predictive Validity - the test score is able to accurately predict performance within some domain that it purports to measure.

Criterion-Related Validity Concurrent Validity Concurrent Validity is established through the use of correlation. “The scores on the Aggressiveness Scale correlated.70 with the rating of teachers of students in their classes.” “The scores on the Beck Hopelessness Scale correlated.81 with the Beck Depression Inventory.”

Criterion-Related Validity Predictive Validity If a test is used to estimate criterion scores at some point in the future, predictive validity must be established. Predictive Validity - the test score is able to accurately predict performance within some domain that is purports to measure. The SAT score, for example, that one establishes in high school is used to predict performance level during one’s first year performance level in college.

Criterion-Related Validity Predictive Validity Some Examples: The SAT Verbal scores have been shown to correlate (.40) with first year students’ grade point average at the end of the first semester at SJC. The Spatial Relations Scale correlated.70 with success in the mental fabricating training for technical high school students.

Criterion-Related Validity Predictive Validity Aptitude and intelligence tests offer evidence regarding predictive validity. When one looks at these tests it is a good idea to examine the utility of predictive tests. Below is a model for considering predictive tests False Positive Positive Negative False Negative

Criterion-Related Validity Concurrent and Predictive Validity Concurrent and Predictive Validity To establish either concurrent or predictive validity it is critically important to make certain that valid, reliable, and relevant measures of the criterion are used.

Construct Validity Construct - is a mental construction of some grouping of variables or behaviors (e.g. anxiety, locus of control, cognitive ability). Construct Validity - establishes that a test score expresses and accurate measure the construct in question.

Construct Validity Because constructs are mental constructions we must derive Construct Validity This is done by examining sets of evidence to build a case for establishing the test's validity. Convergent Validity - high positive correlation with other measures of the same construct Discriminate Validity - low correlation with other measures of a different construct

Multitrait-multimethod Approach to Construct Validity Multitrait-multimethod Approach 1) compute r for same measure of same trait same method 2) compute r for same measure of same trait different method 3) compute r for measure of different trait same method 4) compute r for same measure of different trait different method

Multitrait-multimethod Approach to Construct Validity - An Example Lets suppose we want to establish construct validity for a new depression scale 1) compute r for measure with one that measures the same trait - Beck Depression Scale 2) compute r for measure with a different measurement method same trait - Clinical Interview 3) compute r for measure with one that measures a different trait - Snyder’s Hope Scale 4) compute r for measure with a different trait (Hope) and different method - Clinical Interview

Awareness and Orientation Codes of ethical standards remind test administrators of their responsibility for the orientation of takers. Orientation should describe the purpose of testing, content areas measured, method of administration as well as reporting and use of scores

7 topics to cover in orientation Purpose Criteria used for selecting the test Conditions under which test to be taken Range of skills or domains to be measured Administrative procedures and concerns Types of questions on test and overview Type of scoring, method, + schedule for reporting

Administering the Test Important to deliver instructions in test manual as they are stated. Follow the sequence and timing of the instructions so that your client gets that same information as those in the group on which the test was that normed. Goal for the results to give a valid picture of attributes measured. Examiner should record critical Incidents that deviate from normal conditions.

Posttesting Procedures Recording Test Behavior – checklists/behaviors Major Issues and Problems in Test Administration awareness/orientation phase very important in eliminating probs Examiner and Bias – communication skills/attitudes + expectations/gender/competence/test ethics = examiner bias Feedback on Test and Test Administration – Ask test taker.. how would you rate physical environment, performance, comfort with the administration, time allotted adequate, orientation helpful, fair test?

Communicating Results to Clients, Parents, and Professionals Guidelines & Standards for Communicating Test Results Know test manual, limits of test, informed consent procedures, protect rights Postinterview opportunity to deal w/ interpretation and its use in planning and decision making

Methods for Reporting Test Results 5 major methods Individual Sessions: discussion Group Sessions Written reports Interactive Approaches Video Approaches: current trend

Areas of Consideration Acceptance: goal of feedback session is acceptance of test results and incorporate info into decision making Readiness of client: critical factor in acceptance = client readiness Negative results Flat Profiles: look at other things, interests, values, goals, etc. Motivation and Attitude: more significant when motivated

Scoring Tests Primary models for scoring tests Cumulative – Number of items endorsed Class – Serves to categorize or describe the person Ipsative – How a person performed on a set of variables Alternate & Authentic Assessment Holistic Scoring –individual judgment with model answers

Interpreting Tests Criterion-Referenced Tests describes the specific types of skills, tasks, or knowledge that the test taker can demonstrate. Norm-Referenced Tests compare each person’s score with that of the norm established for that test.

Norms What establishes a norm? Answer: Norms are based on the occurrence of the majority, or bulk, of values within a distribution obtained from some defined sample of individuals (Think of the Normal or Bell Curve). Norms are used to give information about performance relative to what has been observed within a sample.

Age-Related Norms Some normative groups have been established for particular ages. Once age-related norms have been established tracking is possible. Tracking is a process by which, with some level of confidence, one can expect to see specific characteristics in a sample of interest. Examples: Developmental activities of infants, Grade based reading levels, Age based capacity for task of increasing cognitive complexity

Within-Group Norms Establishing norms is very easy to do. It is important to keep mindful that the science of testing is not perfect. One must be cautious, therefore, about assuming that a test is a valid measure for everyone taking it. If a test provides an accurate measure for only some members of a population accuracy of within-group norms must be questioned.

Percentile, Percentile Ranks, Quartiles, and Deciles Percentiles - divide the total frequency for a set of observations into hundredths Percentile Rank - establishes that point below which a certain percent of scores fall Quartile - divide the total frequency for a set of observations into quarters Q1 = 25% Q2 = 50% Q3 = 75% Q4 = 100% Deciles - divide the total frequency for a set of observations into tenths

Standard Scores One of the problems with establishing a mean and standard deviation for a specific distribution is the limitations to the meaning of such measures. Standard scores are a means of presenting the relative position of an individual on a test that are tied to the normal curve. There are two major forms of standard scores that you will encounter. The Z Score and the T Score

Z Scores Z Scores are based on an mean of 0 and a standard deviation of 1. Z Score Table - Appendix 1 - Salkind x - X Z = s Z = =

Z Scores and the Normal Curve Z Score Table - Appendix 1 in Salkind Z Scores give us a opportunity to establish a better understanding of where individual scores occur relative to standardized norms.

T Score T Scores - are standard scores using a fixed mean and standard deviation in units that eliminate the need for decimals and signs. On many tests the arbitrary mean or fixed mean is 50 and the arbitrary standard deviation is 10. T = s(z) + { X} T = 10(-.50) + 50 T = T = 45