1. Appraisal and Its Application to Counseling COUN 550 Saint Joseph College Ability, Intelligence, Aptitude and Achievement Testing For Class #12 Copyright © 2005 by R. Halstead. All rights reserved.

2. Class Objectives Assessment Issues in Education / Schools High-Stakes Testing & Federal Law Standards for Practice Student Presentations & Review for Exam

3. Testing in Schools Criterion-referenced academic achievement tests (also called minimum-level or essential skills tests) that are used to make important evaluations and decisions are often called ‘high-stakes testing’ No Child Left Behind Act of 2001 Accountability

4. Professional Standards Responsibilities of Users of Standardized Test or RUST (AACE) Standards for Educational and Psychological Testing – often used as legal reference in court cases. (American Educational Research Association, APA, & National Council on Measurement in Education) Code of Fair Testing Practices (Joint Committee on Testing Practices)

5. School Counselors Percentage of school counselors reporting they are highly confident of their ability to: understand and use information such as norms and standardized scores (64%) use statistics such as the mean, standard deviation, or correlation (57%) use error of measurement information (51%) use test reliability and validity information (48%).

6. School Counselors Commonly Use: group achievement tests 71% group aptitude-ability tests 57% interest inventories 55% individual aptitude-ability tests 50% vocational-personal preference tests 45%

7. A Note of Caution If you have any choice at all, do not try to construct your own measures. Why? So far we have just scratched the surface in addressing reliability and validity. Constructing a valid and reliable measure involves a long and difficult process of statistical analysis requiring many resources. Best Practice: Fine published measures that have validity and reliability data.

8. Throwing Caution to the Wind Even though many measures are available for a variety of purposes, unique situations and/or populations, at some point in your career you will most likely be asked to construct a measure. Why? A hospice wishes to measure the level of satisfaction family members experience with agency programs. A child guidance clinic wants to establish its approachability by surveying the adolescents who terminated early in counseling against advice.

9. Throwing Caution to the Wind So, if you must construct a measure - be sure that you write the best items possible. Define exactly what you want to measure Consult relevant literature to guide your efforts Avoid long wording of items (one element/idea per item) Gage the complexity of vocabulary to the population Generate an item pool that mixes response task to avoid an acquiescence response set Write items that are appropriate to population, language and culture

10. Item Formats The Dichotomous Format - offers two alternatives for each item Example: True/False, Yes/No, Agree/Disagree, etc. Advantages: ease in writing and scoring Disadvantages: 50% - 50% chance in guessing The Polytomous Format - more than two alternatives Example: Multiple choice Advantages: ease of scoring, cover a large amounts of material, reduces the chances of successful guessing Disadvantages: can still guess

11. Item Formats The Likert Format - requires a response regarding the degree of agreement. Example: Strongly Disagree, Moderately Disagree, Mildly Disagree, Mildly Agree, Moderately Agree, Strongly Agree The Category Format - Similar to the Likert but there is number scaling and the polls of the scale are labeled. Example: Subject Units of Discomfort Scale (SUDS)

12. Item Formats Checklists - requires a response to a list to trigger descriptions or adjectives Example: Symptom Checklist 90-R Q-Sort - Require the participant to sort relevant statements in to a specified number of piles that range according to level of agreement.

13. Item Analysis and Discriminability Item analysis is an important part of test construction in that it determines the degree to which each item actually measures an aspect of the construct that you what it to measure. In this way it is closely related to validity of the test. Item disciminability determines whether the item holds true to the results on the whole test

14. Selection and Decision Analysis Test selection is critically important. You should keep a mental checklist as you go through and test manual or, in the the case of a nonproprietary test, the journal article in which it is published Sampling Reliability and Validity Scoring Practical Considerations