Standardization and Test Development Nisrin Alqatarneh MSc. Occupational therapy

WHY SHOULD THERAPISTS USE STANDARDISED TESTS?  Most testing errors occur because therapist:  collected insufficient data  collected inaccurate data  used unsystematic data-collection methods  obtained irrelevant data  failed to verify data collected  obtained data that have been contaminated by bias or prejudice (e.g. either by the person completing the self-report or proxy survey or by the therapist)  not accurately communicated data collected

 A rigorously developed and psychometrically robust test will:  involve the collection of sufficient data for the purpose of the test (e.g. to screen for a specific impairment)  have established reliability so data are collected accurately  use a systematic data-collection method  have established validity so that data obtained are related to the stated purpose and focus of the test  provide information about confidence intervals so the therapist can judge how likely it is that this test result has provided a true picture of the person’s ability and/or deficits  reduce the influence of bias or prejudice on test results  have a record form for recording, analysing and communicating scores.

 The major advantage of a statistically-based assessment – and it is crucial to professional effectiveness and quality of decision making – is that it provides an objective source of data, unbiased by subjective feeling  Definition of objective: Facts or findings which are clearly observable to and verifiable by others, as reflecting reality.  Definition of subjective An observation not rigidly reflecting measurable reality; may imply observer bias; may not be verifiable by others in the same situation.

WHAT IS A STANDARDISED TEST?  The AOTA defines the word standardised as:  ‘made standard or uniform; to be used without variation; suggests an invariable way in which a test is to be used, as well as denoting the extent to which the results of the test may be considered to be both valid and reliable’

What makes a test standardized?  A measurement tool that is published  Has been designed for a specific purpose for use with a particular population  Should have detailed instructions explaining how and when it should be administered and scored and how to interpret scores (a test protocol).  It should also present the results of investigations to evaluate the measure’s psychometric properties  Details of any investigations of reliability and validity should also be given.  The conditions under which standardised tests are administered have to be exactly the same

WHAT IS AN UN-STANDARDISED ASSESSMENT?  Are assessments that provide the therapist with information but have no precise comparison to a norm or a criterion.  Some un-standardized tests are structured assessments that are constructed and organized to provide guidelines for the content and process of the assessment, but their psychometric properties have not been researched and documented

STANDARDISATION  A process of taking an assessment and:  Developing a fixed protocol for its administration and scoring  And conducting psychometric studies to evaluate whether the resultant assessment has acceptable levels of validity and reliability.  There are two ways in which assessments can be standardized: 1. In terms of procedures, materials and scoring 2. In terms of normative standardization

To standardize the test materials:  The exact test materials should be listed or included (precise details of how to construct the test -with exact sizes,colours, fabric of materials etc.- are required)  OR, provide materials as part of a standardized test battery

To standardize the method of administration:  The test conditions should be described in detail  The number of people tested at any one time should be specified (individual or group)  Information about time required for administration should be given  Detailed written instructions for the therapist should be provided  Exact wording for any instructions to be given to the test taker are required

Example:

To standardize the scoring system:  Clear guidelines need to be provided for scoring  Scoring methods vary from test to test and may include the use of raw scores that may then be converted to another type of score  Information should be included to guide the therapist in the interpretation of scores  Therapists should be aware that a number of factors may influence the person’s performance on the test, including test anxiety, fatigue, interruptions and distractions

CONSTRUCTING A STANDARDISED TEST  A 10-step test construction process: 1. Identify the primary purpose(s) for which test scores will be used. 2. Identify behaviours that represent the construct or define the domain. 3. Prepare a set of test specifications, delineating the proportion of items that should focus on each type of behaviour identified in step two. 4. Construct an initial pool of items. 5. Have items reviewed (and revise as necessary). 6. Hold preliminary tryouts (and revise as necessary). 7. Field-test the items on a large sample representative of the examinee population for whom the test is intended. 8. Determine statistical properties of item scores and, when appropriate, eliminate items that do not meet pre-established criteria. 9. Design and construct reliability and validity studies for the final form of the test. 10. Develop guidelines for test administration, scoring and for the interpretation of test scores (e.g. prepare norm tables, suggest recommended cutting scores or standards for performance etc.).

CRITERION-REFERENCED TESTS  Is a test that examines performance against pre- defined criteria and produces raw scores that are intended to have a direct, interpretable meaning  The person’s score on this type of test is interpreted by comparing the score with a pre-specified standard of performance or against specific content and/or skills  Therapists use this type of test to judge whether the person has mastered the required standard (for example to judge whether the person has sufficient independence to be safely discharged from hospital to live alone at home)  The person’s performance is NOT compared to the ability of other people

CRITERION-REFERENCED TESTS  Criterion-referenced tests usually have one of two main purposes, which are:  Estimation of the domain score (i.e. the proportion of items in the domain that the subject can pass correctly)  Mastery allocation (The observed test results are used to classify people into the mastery categories).  Criterion-referenced tests are important because therapists are concerned with desired outcomes  In a therapy assessment a score representing master might be labelled able or independent or observed,  while a score representing non-master might be labelled unable or dependent or not observed.

NORM-REFERENCED TESTS  A person’s performance can be considered in the light of expected, or ‘normal’, performance by comparing it to the range of scores obtained by a representative group of people  Norms is defined as ‘sets of scores from clearly defined samples’  Norms indicate the average performance on the test and the varying degrees of performance above and below average  Norms should be based on a large, representative sample of people for whom the test was designed  The normative sample should reflect the attributes of the client population

CRITERIA FOR JUDGING NORMS  Norms should be relevant to your area of practice  The normative sample should be representative of your client group  Norms should be up to date  Comparable (If you want to compare scores on tests with different norm groups)  Norms should be adequately described

Any Questions? Assessment in OT