Computer Applications in Testing and Assessment James P. Sampson, Jr. Florida State University Copyright 2002 by James P. Sampson, Jr., All Rights Reserved

2 Overview of Topics Computer-Assisted vs. Computerized Rationales for Computer-Assisted Testing Potential Benefits Potential Problems Future Use

3 Computer-Assisted vs. Computerized Both describe use of computer technology in assessment process Used with nearly the same frequency Using the term computerized may cause problems and foster misunderstanding

4 Computerized Implies that computers replace humans previously needed to complete a task May lead to practitioner under use or misuse An inaccurate depiction of computer technology in testing

5 Computerized Under Use –Uninformed rejection of computer technology –View computers as performing inappropriate functions –Believe computers lead to dehumanization

6 Computerized Misuse –View computers as effective in performing functions not intended by developer –Use of computers to compensate for a lack of professional skill

7 Computer-Assisted Computer fulfills a support role Improves, not replaces the human element More likely to avoid under use or misuse

8 Rationales for CAT 1.Focus on assigning humans and computers to tasks best suited to the capabilities of each 2.Develop previously unavailable techniques to improve the quality of tests and assessments

9 Potential Benefits of CAT Overview of topics: –Enhanced validity and reliability –Enhanced client integration of assessment data in counseling –Enhanced staff efficiency and cost-effectiveness

10 Enhanced Validity and Reliability Overview of topics: –Test Administration and Scoring –Administration of Interviews and Checklists –Computer-Based Test Interpretation

11 Test Administration and Scoring Greater degree of standardization More accurate results for people with disabilities Availability of adaptive testing Collection of ancillary data (e.g., response manner and specific response recording)

12 Enhanced Validity and Reliability Overview of topics: –Test Administration and Scoring –Administration of Interviews and Checklists –Computer-Based Test Interpretation

13 Administration of Interviews and Checklists More comprehensive topic coverage More consistent topic coverage Less evaluative tension from repeated questioning by the practitioner Results less affected by time of day More honest client response

14 Enhanced Validity and Reliability Overview of topics: –Test Administration and Scoring –Administration of Interviews and Checklists –Computer-Based Test Interpretation

15 Computer-Based Test Interpretation (CBTI) Assists in the interpretation of test data Provides expanded & consistent knowledge base Includes research data and the clinical experience of other clinicians Standardizes computer functioning Provides consistency

16 Types of CBTI Descriptive interpretations Clinician-modeled interpretations (renowned clinician type) Clinician-modeled interpretations (statistical model type) Clinical actuarial interpretations

17 Descriptive CBTI Descriptive sentences related to specific scales Scales interpreted independent of each other Typically cryptic in nature

18 Clinician-Modeled CBTI (Renowned Clinician Type) Replicates the interpretive judgment of a single clinician Based on the clinician with the most experience with the test Provides consistent results Mistakes also consistent

19 Clinician-Modeled CBTI (Statistical Model Type) Replicates interpretive judgments of a group of practitioners Includes a group comprised of experienced clinicians Statistically valid model Incorporates the best consensus possible among experts

20 Clinical Actuarial CBTI Provides narrative descriptions and clinical hypotheses Based on clinical research findings for particular score patterns

21 Potential Benefits of CBTI More comprehensive and objective Not subject to interpreter bias or extraneous circumstances Provides rapid data processing Ready for use in early counseling sessions Helps practitioners organize and systematically access extensive data bases Easily updated when new research is available

22 Potential Benefits of CBTI CBTI serves in a consulting role Practitioner integrates data from variety of sources to create assessment reports Computerized interpretive reports only used as adjuncts, not as the final word

23 Potential Benefits of CAT Overview of topics: –Enhanced validity and reliability –Enhanced client integration of assessment data in counseling –Enhanced staff efficiency and cost-effectiveness

24 Enhanced Client Integration of Assessment Data in Counseling Computer applications used as one aspect of the therapeutic process CBTI reports in clinical settings are typically not given to most clients Interpretive reports are typically provided directly to clients in career counseling –provides basic understanding of results –outlines the decision-making process –suggests follow-up resources

25 Enhanced Client Integration of Assessment Data in Counseling Generalized test interpretations –video-based technology

26 Potential Benefits of CAT Overview of topics: –Enhanced validity and reliability –Enhanced client integration of assessment data in counseling –Enhanced staff efficiency and cost-effectiveness

27 Enhanced Staff Efficiency and Cost-Effectiveness Computer-assisted testing is more cost- effective Saves testing time Improves time utilization of support staff Gives practitioners more time to prepare test interpretations, write test reports, and work directly with clients on more complex issues

28 Potential Problems Associated with Computer Applications Overview of Topics –Threats to Validity and Reliability administration and scoring test interpretation ethical issues ethical standards –Ineffective Implementation

29 Threats to Validity and Reliability Administration and Scoring –Norms used from paper-and-pencil tests –Blanket assumptions of equivalency –Scoring errors –Perceived inherent credibility - infallibility –Client misunderstandings of instructions

30 Threats to Validity and Reliability Test Interpretation –Different reports from different software developers –Differences in CBTI reports and conclusion reached by clinical interviewer –Computer may be perceived adding a level of scientific precision that does not exist

31 Threats to Validity and Reliability Misuse of CBTI –Qualified practitioners may misuse CBTI as a result of being overworked –Unqualified practitioners may use to offset their lack of training and experience –Practitioner dependence on CBTI

32 Threats to Validity and Reliability Appropriate Use of CBTI –Computer serves as a consulting function –Practitioner has final responsibility for communicating test results –Qualifications necessary for appropriate use –Prescreening –Not all clients can provide valid item responses –Follow ethical standards

33 Potential Problems Associated with Computer Applications Overview of Topics –Threats to Validity and Reliability administration and scoring test interpretation ethical issues ethical standards –Ineffective Implementation

34 Ineffective Implementation Staff Resistance –Lack of staff participation in decision-making –Lack of experience with computers –Poor quality program documentation –time and effort required Poor staff support results in reduced acceptance

35 Effective Implementation Improves the likelihood of success –Careful planning –Group input in decision making –Reasonable rate of change Effective implementation involves interpersonal dynamics more than technology

36 The Future Hardware Development –Human voice –Video-based stimuli –Virtual reality Possible Problems –High cost –Extraneous auditory and visual data may distract test taker –Stereotypes

37 The Future Software Development –Expert and decision support systems –Expert systems will be added to existing CBTI systems –Expert systems can be used as a powerful training resource

38 Conclusion Do computer applications actually improve testing and assessment in counseling? –Depends more on practitioner attitudes and skills than future advances in technology –Without effective use, new technology is meaningless