1. EPSY 625 LECTURE 3 COGNITIVE ASSESSMENT

2. AFFECT

3. TASK DEMANDS: STRUCTURING COGNITIVE TESTS TYPES ARTIFICIAL ANALOG ACTUAL TESTS: 1. ACHIEVEMENT 2. INTELLIGENCE

4. TASK TYPES ARTIFICIAL- intended to assess response to novel conditions not encountered before ANALOG- intended to assess response to conditions not ethically or economically establishable ACTUAL- intended to assess response in “real” setting

5. I. ACHIEVEMENT TESTS A. Curriculum Guides/ Mandates: TEXAS ESSENTIAL KNOWLEDGE AND SKILLS (TEKS)  TAKS B. Textbook Surveys C. Political/Philosophical Selection (e.g.. Hirsch, “CULTURAL LITERACY) D. Myth/Nostalgic/History “all _______ should know this

6. ACHIEVEMENT TESTS D. Myth/Nostalgic/History “all _______ should know this” E. Issue: content/pedagogical validity F. Content selection Random/Ordered Importance: how determined? Taxonomies

7. TAXONOMIES a)Bloom et al. Evaluation Synthesis Analysis Application Comprehension Knowledge

8. b)Table of specifications Topics Taxonomy KCApAn A699630% B1015151050% C466420% 20%30%30%20%100%

9. SAMPLING OF BEHAVIORS All tests sample a Universe, defined by a combination of all possible tasks, occasions, raters, and measurement methods Domain refers to a content area to which the tasks refer Sampling Variability of Performance Assessments Richard J. Shavelson, Gail P. Baxter, and Xiaohong Gao Journal of Educational Measurement, Fall 1993, Vol. 30, No. 3, pp. 215-232

10. COGNITIVE RESEARCH 1. Memory effects - STM/LTM or Level: working-long term 2. Processing effects - spatial - analogical - reasoning - integrative/simultaneous

11. COGNITIVE RESEARCH 3. hot cognition/affect/ motivation - Paivio’s “dual coding” theory 4. Task structure - VISUAL/SPATIAL - SEMANTIC/VERBAL - PROCEDURAL/ORDER

12. COGNITIVE RESEARCH 5. Knowledge structure - Declarative (what) - Procedural(how-strategy) - Conditional (when-strategy)

13. CONTENT SELECTION IMPLICATIONS Situational nature of performance Complexity in development Limitations in generalizability

14. II. INTELLIGENCE TESTS A. THEORY BASIS: 1. “g” Construct- single factor 2. Limited # (2 or 3 factors) 3. Multiple intelligences 4. Limited scope for assessment (school)

15. 1. “g” FACTOR BINET-TERMAN: children’s mental functioning: STANFORD-BINET IQ British psychology: SPEARMAN: factor analysis RAVEN: Progressive Matrices

16. 2. Limited # of factors WECHSLER and adults: Verbal and non- verbal IQ: WAIS WAIS-R Developmental downward extension: WISC WISC-R WISC-III

17. 2. Limited # of factors KAUFMAN & KAUFMAN: KAB-C: children’s IQ simultaneous & sequential (from Luria’s cognitive theory)

18. 3. MULTIPLE INTELLIGENCES MERCER’S SOMPA (System of Multicultural Pluralistic Assessment) GARDNER’s Multiple Intelligences STERNBERG’s Analogical Reasoning Theory Subtest use of SB, WISC-III, and KAB-C

19. 4. LIMITED SCOPE School intelligence (Publisher developed): Differential Aptitude Tests Scholastic Achievement Tests: SAT, GRE, GMAT COGAT (Cognitive Abilities Test), etc.

20. B. SAMPLING OF PROCESSES SPECIFIC ABILITIES/PROCESSES: ANALOGICAL REASONING SPATIAL ABILITY MEMORY NONVERBAL: BLOCK DESIGN

21. TYPES OF RESPONSE I. SUPPLY II. SELECT

22. I. SUPPLY RESPONSES A. WRITTEN (ESSAY, SHORT ANSWER) B. ORAL C. DRAWING/SKETCHING D. COMPUTATION E. PERFORMANCE

23. A. WRITTEN- ESSAY Extended response allows greater sampling of knowledge domain Dependent on writing (computer vs. pen) speed, legibility, strategic knowledge Greater time to score- need for rubric, fatigue in scoring Need to constrain topic, task

24. A. WRITTEN- SHORT ANSWER Restrict topics: Definitions or concepts Quick computations Limit response length Establish protocol for scoring Establish scoring system

25. B. ORAL RESPONSE Historical precedence (Greek, Roman, European Middle Ages, University system) Performance aspect: knowledge and personal interaction Typically faster response required- “thinking on one’s feet”

26. C. SKETCHING/DRAWING Less commonly required Often task-specific (e.g.. Knowledge maps) Incorporated into broader assessments or tasks (e.g.. Part of physics or math problem)

27. D. COMPUTATION Most common to mathematics and science fields Mental or written requirement Use of calculators or computers Verbal component may be important (word problems) Spatial component may be important (imageability)

28. PERFORMANCE COMPLEX CONSTELLATION OF ACTIVITIES SIMULATION REAL SITUATION ISSUES

29. PERFORMANCE ISSUES COST TIME TO SET UP TIME TO SCORE- RELIABILITY # OF TASKS SAMPLED AUTHENTICITY VS. VALIDITY

30. II. SELECTION RESPONSE MULTIPLE CHOICE (INCLUDING T-F) MATCHING

31. MULTIPLE CHOICE EFFICIENT SAMPLING- Time, cost RELIABILITY- produces reliable measures VALIDITY ISSUE: Does selection represent same knowledge as supply? Limitations overstated by critics current development does not take advantage of information available, new cognitive theory

32. MULTIPLE CHOICE Livingston, Reynolds & Willson (2005) list for good item writing # options- depends on good alternatives Options generated from incorrect cognitive processes, become clues to knowledge structure (e.g.. BUGGY arithmetic program)

33. MATCHING VARIANT ON MULTIPLE CHOICE LIMIT # OF MATCHES < 10 REQUIRE SINGLE CONCEPT REQUIRE MORE OPTIONS THAN QUESTIONS