Chapter 5 Instrument Selection, Administration, Scoring, and Communicating Results

Selection of an Assessment Instrument  Identify possible alternatives Educational Testing Services may be useful Educational Testing Services Mental Measurement Yearbook (MMY) reviews many tests and is available at Morris or reviews can be ordered on-line for a fee. Mental Measurement Yearbook  Client’s lives can be adversely affected by selecting and using a faulty instrument

Evaluating an Instrument  Test Purpose  Instrument Development  Appropriate Norm Group or Criterion  Reliability  Validity  Bias  Interpretation and Scoring  User Qualifications

Test purpose  Does instrument meet counseling needs?  Manual may state purpose Purpose may not relate to name Purpose may not be met because of construction problems

Instrument development  Construction of items crucial  Look for detailed item analysis data  Face validity

Selection of norm group  Is norming group appropriate for client? Age Gender Ethnicity Socioeconomic representation Educational level Geographic location

Selection of criterion  How did developers determine criterion? Standards from professional organizations Reviews by leading researchers Studies of curriculum  Manual must supply enough data to be convincing

Reliability  No clear guidelines for what constitutes “good” reliability  Type of instrument affects how reliability should be viewed  Can be estimated in different ways and manuals will often provide range of coefficients calculated using different methods

Validity  Look for preponderence of evidence  Manual must provide enough information to judge  GO TO THE LITERATURE!

Bias  Test bias in context of multicultural issues Differential item functioning (item difficulty differs for ethnic groups) Reliability coefficients may differ ethnically Construct validity may be compromised as the construct may be culturally based

Interpretation and scoring materials  Manual must provide adequate description of what scores mean  Hand vs. machine scoring

User qualifications  Level A: manual is sufficient (Holland’s Self-Directed Search)  Level B: master’s degree or equivalent training (Myers-Briggs, Strong Interest Inventory)  Level C: doctoral level or equivalent training (Wechslers, Binet, MMPI-2)  Code of Fair Testing Practices in Education Code of Fair Testing Practices in Education  ACA Code of Ethics, Section ESection E

Practical issues  Cost  Time

Evaluating instruments  Form in text, page 95  Instrument evaluation paper

Administering Assessment Instruments  Pre-read administration materials  Follow standardized procedures

Administrator Effects  Expectancy (pygmalion effect)  Some data support minor effects

Examiner/Examinee Relationship  Some studies support impact of relationship for kids and even more so for lower SES kids  Standardized test manuals usually address relationship boundaries

Examiner race  Research equivocal

Scoring  Hand scoring Clients can score some assessments (SDS) but may be clinical reason not to Some tests require multiple templates  Computer scoring and interpretation Few to no errors Counselors ethically bound to know integrity of service and steps taken to develop scoring and interpretation programs

Performance and Authentic Assessment Scoring  Associated with achievement testing (rather than grammar test, write business letter)  Goal is to see if knowledge can be applied  Big issue is objectivity of scoring

Performance and Authentic Assessment Scoring Guidelines  Assessment has specific focus  Scoring plan is based on observable qualities  Scoring is designed to reflect the intended target  Setting for assessment is appropriate  Checklist or rating scales are used  Scoring procedures have been field tested before they are used

Communicating Results  Must know manual information  Optimize power of test (It’s useful in these ways for these reasons) rather than allowing client to maximize (test speaks truth)  Use effective counseling skills  Develop multiple methods of explaining in “plain” English

Communicating Results (cont’d)  Use descriptive terms rather than numerical scores and tie them to the reason for the assessment  Put results in context of other client information  Involve client in interpretation and ask for feedback often

Communicating Results (cont’d)  Limitations are discussed in non- technical terms  Encourage client to ask questions  Summarize results to iterate and stress important points

Communicating Results to Parents  Be empathic  Monitor parents’ reactions so child does not internalize as blame  Provide therapeutic environment