EPSY 625 LECTURE 3 COGNITIVE ASSESSMENT

AFFECT

TASK DEMANDS: STRUCTURING COGNITIVE TESTS TYPES ARTIFICIAL ANALOG ACTUAL TESTS: 1. ACHIEVEMENT 2. INTELLIGENCE

TASK TYPES ARTIFICIAL- intended to assess response to novel conditions not encountered before ANALOG- intended to assess response to conditions not ethically or economically establishable ACTUAL- intended to assess response in “real” setting

I. ACHIEVEMENT TESTS A. Curriculum Guides/ Mandates: TEXAS ESSENTIAL KNOWLEDGE AND SKILLS (TEKS)  TAKS B. Textbook Surveys C. Political/Philosophical Selection (e.g.. Hirsch, “CULTURAL LITERACY) D. Myth/Nostalgic/History “all _______ should know this

ACHIEVEMENT TESTS D. Myth/Nostalgic/History “all _______ should know this” E. Issue: content/pedagogical validity F. Content selection Random/Ordered Importance: how determined? Taxonomies

TAXONOMIES a)Bloom et al. Evaluation Synthesis Analysis Application Comprehension Knowledge

b)Table of specifications Topics Taxonomy KCApAn A699630% B % C466420% 20%30%30%20%100%

SAMPLING OF BEHAVIORS All tests sample a Universe, defined by a combination of all possible tasks, occasions, raters, and measurement methods Domain refers to a content area to which the tasks refer Sampling Variability of Performance Assessments Richard J. Shavelson, Gail P. Baxter, and Xiaohong Gao Journal of Educational Measurement, Fall 1993, Vol. 30, No. 3, pp

COGNITIVE RESEARCH 1. Memory effects - STM/LTM or Level: working-long term 2. Processing effects - spatial - analogical - reasoning - integrative/simultaneous

COGNITIVE RESEARCH 3. hot cognition/affect/ motivation - Paivio’s “dual coding” theory 4. Task structure - VISUAL/SPATIAL - SEMANTIC/VERBAL - PROCEDURAL/ORDER

COGNITIVE RESEARCH 5. Knowledge structure - Declarative (what) - Procedural(how-strategy) - Conditional (when-strategy)

CONTENT SELECTION IMPLICATIONS Situational nature of performance Complexity in development Limitations in generalizability

II. INTELLIGENCE TESTS A. THEORY BASIS: 1. “g” Construct- single factor 2. Limited # (2 or 3 factors) 3. Multiple intelligences 4. Limited scope for assessment (school)

1. “g” FACTOR BINET-TERMAN: children’s mental functioning: STANFORD-BINET IQ British psychology: SPEARMAN: factor analysis RAVEN: Progressive Matrices

2. Limited # of factors WECHSLER and adults: Verbal and non- verbal IQ: WAIS WAIS-R Developmental downward extension: WISC WISC-R WISC-III

2. Limited # of factors KAUFMAN & KAUFMAN: KAB-C: children’s IQ simultaneous & sequential (from Luria’s cognitive theory)

3. MULTIPLE INTELLIGENCES MERCER’S SOMPA (System of Multicultural Pluralistic Assessment) GARDNER’s Multiple Intelligences STERNBERG’s Analogical Reasoning Theory Subtest use of SB, WISC-III, and KAB-C

4. LIMITED SCOPE School intelligence (Publisher developed): Differential Aptitude Tests Scholastic Achievement Tests: SAT, GRE, GMAT COGAT (Cognitive Abilities Test), etc.

B. SAMPLING OF PROCESSES SPECIFIC ABILITIES/PROCESSES: ANALOGICAL REASONING SPATIAL ABILITY MEMORY NONVERBAL: BLOCK DESIGN

TYPES OF RESPONSE I. SUPPLY II. SELECT

I. SUPPLY RESPONSES A. WRITTEN (ESSAY, SHORT ANSWER) B. ORAL C. DRAWING/SKETCHING D. COMPUTATION E. PERFORMANCE

A. WRITTEN- ESSAY Extended response allows greater sampling of knowledge domain Dependent on writing (computer vs. pen) speed, legibility, strategic knowledge Greater time to score- need for rubric, fatigue in scoring Need to constrain topic, task

A. WRITTEN- SHORT ANSWER Restrict topics: Definitions or concepts Quick computations Limit response length Establish protocol for scoring Establish scoring system

B. ORAL RESPONSE Historical precedence (Greek, Roman, European Middle Ages, University system) Performance aspect: knowledge and personal interaction Typically faster response required- “thinking on one’s feet”

C. SKETCHING/DRAWING Less commonly required Often task-specific (e.g.. Knowledge maps) Incorporated into broader assessments or tasks (e.g.. Part of physics or math problem)

D. COMPUTATION Most common to mathematics and science fields Mental or written requirement Use of calculators or computers Verbal component may be important (word problems) Spatial component may be important (imageability)

PERFORMANCE COMPLEX CONSTELLATION OF ACTIVITIES SIMULATION REAL SITUATION ISSUES

PERFORMANCE ISSUES COST TIME TO SET UP TIME TO SCORE- RELIABILITY # OF TASKS SAMPLED AUTHENTICITY VS. VALIDITY

II. SELECTION RESPONSE MULTIPLE CHOICE (INCLUDING T-F) MATCHING

MULTIPLE CHOICE EFFICIENT SAMPLING- Time, cost RELIABILITY- produces reliable measures VALIDITY ISSUE: Does selection represent same knowledge as supply? Limitations overstated by critics current development does not take advantage of information available, new cognitive theory

MULTIPLE CHOICE Livingston, Reynolds & Willson (2005) list for good item writing # options- depends on good alternatives Options generated from incorrect cognitive processes, become clues to knowledge structure (e.g.. BUGGY arithmetic program)

MATCHING VARIANT ON MULTIPLE CHOICE LIMIT # OF MATCHES < 10 REQUIRE SINGLE CONCEPT REQUIRE MORE OPTIONS THAN QUESTIONS