LECTURE 13 EPSY 625 TEST BIAS ITEM BIAS ITEM BIAS DETECTION

TEST BIAS Issue arose after civil rights legislation in the 1960s African-americans (and other minorities) were either denied jobs or placed in lower category/paying jobs based on test scores

TEST BIAS Definition developed over time: related to disparities in selection different from base rates of application or of population: 50% of applicants were minority 10% hired were minority Prima facie evidence of discrimination- may be legal (eg. Female models for women’s clothes)

TEST BIAS Illegal discrimination based on series of legal cases Supreme court review of the issues separated Hiring and economic discrimination Education

LEGAL CASES INVOLVING TESTING

Griggs v. Duke Power (Supreme Court of the U.S.,1971) Tests used for selection of employees may not discriminate unfairly in form and operation: the test must be shown to be job-related, specifically a relation to successful job performance.

Albemarle Paper Company vs. Moody (Supreme Court of the U.S. 1975) A test used for selection of employees must validated with a population similar to that for whom it is used. Awards for improper conduct were given.

United States vs. State of South Carolina (U.S. District Court, South Carolina, 1977) Content of the National Teacher Examination was upheld as adequately validated when disparate impact on certification of black and white teachers was observed. Overturned criterion-related validity as the only standard for validation.

Diana v. California State Board of Education, No. C RFP, District Court of Northern California, Ruled for plaintiff that IQ testing for special education populations must be normed on culturally and linguistically relevant populations and contain no culturally biased materials.

Golden Rule Insurance Company v. Washburn, Illinois Circuit Court, 7 th Ind. Cir. Ct. (1984). Consent Decree. Ruling that any item on a test that separates protected groups is biased, items with least difference are to be selected. APA formally disagreed.

Larry P. v. Riles, 343 F. Supp (N.D. Cal. 1972) (preliminary injunction), affirmed 502 F. 2d 963 (9 th Cir. 1974), opinion issued No. C RFP (N.D. Cal. Ocotber 16, 1979). Ruling for plaintiff that IQ tests are biased.

PASE v. Hannon, 506 F. Supp. 831 (N.D. Ill. 1980) Ruling against plaintiff that IQ tests are not biased.

ITEM BIAS Test development was criticized for generating test questions that were systematically biased against minorities and women Wording/phrasing Pictures/illustrations Scoring Numerous examples of bias were shown on current tests (1960s-70s)

ITEM BIAS Technical criteria were developed under the term “item bias” Changed to Differential Item Functioning (DIF) to account for any item that performed in an inconsistent manner for different groups - Did not require specific theoretical or conceptual argument to remove item - Subtle bias might not be easily observed - Empirical effects differed from “apparent” bias as determined by minority experts

ITEM BIAS DETECTION IRT BASED – requires sophisticated software, reserved for commercial applications or expert researchers CHI-SQUARE BASED- relatively easy to use, potential with small samples and clinical samples ANOVA BASED- theoretically linked but not empirically very good

IRT - BASED ITEM BIAS Find D, where D=max[p(  maj ) -p(  min )] D min maj

CHI-SQUARE BASED MANTEL-HAENSZEL LOG-ODDS: RELATIVE PROPORTIONS CORRECT RESPONSES FOR EACH ABILITY CATEGORY SCHEUNEMANN CHI SQUARE FOR INTERACTION OF GROUP (MIN-MAJ) BY ABILITY CATEGORY

MANTEL-HAENSZEL ITEM PROCEDURE 20%30% 20% 30% 25%15% MAJ MIN QUARTILES ON TOTAL TEST Differences in proportion are cumulated into a chi-square statistic with 3 df

ANOVA-BASED ANCOVA BASED: INTERACTION OF GROUP (MIN-MAJ) x COVARIATE (ABILITY) 0-1 ABILITY 1010 Mean difficulty across ability/achievement scores maj min