1. RTTT 1 Innovation and Technology in Assessment: Comments Eva L. Baker Director National Center for Research on Evaluation, Standards, & Student Testing UCLA Public and Expert Input Meeting, Race to the Top, U.S. Department of Education Boston, MA November 12-13, 2009 © Regents of the University of California

2. RTTT 2

3. RTTT 3 Overview: BIG challenges underlying innovative tasks Stimulate multi-level, multi-purpose system, open architecture, open source, scalable, top down- bottom up Invest in design not just administration Translate standards into rich representations of content, cognitive demands, situations and their relationships tuned for summative, interim, and formative uses Design assessments to support learning

4. RTTT 4 Big Challenges Obtain validity evidence, incentives for using new methods Exploit new options, games, immersive environments, simulations only if necessary for assessment information, anticipate change, SCORM Design to cost, with incentives for economies Save $ using authoring systems, reusable templates and objects for continued renewal & amortization of system Train users Find mid point between page-turning and dazzle

5. RTTT 5 1. Criteria for Innovative Tasks (other than efficiency) Emphasis on priority attributes of standard (s) in multi-step, realistic situations Clear expectations Teachable (or learnable), not trait-based Opportunity for multiple paths to solution or product Componential, reusable objects to permit quick creation of similar tasks Tagged to key dimensions in database Explicit feedback and areas for additional effort given

6. RTTT 6 1. Technology Features – off the shelf Hand-held for place-based learning, networking, collaboration,and teamwork, homework or informal assessment, feedback for teaching Sensors to permit mapping of movement, engagement in tasks, screen time Distraction, e.g., irrelevant animation Monitor student activity, feedback, revision processes to guide instruction Voice and handwritten inputs to broaden use Potential rrelevant inferences

7. RTTT 7 K Mapper Student Work

8. RTTT 8 1. K Mapper Student Action

9. RTTT 9 1. K Mapper Expert Scoring Model

10. RTTT 10 1. Rescuing Animals Using Ratios and Equations

11. RTTT 11 1. PuppetMan Math Problem-solver © Regents of the University of California

12. RTTT 12 2/3/4/ Focusing on first elements: essential design components Graphical representation of standards Content elements & relationships Cognitive demands Task situations Linguistic requirements Needed prior knowledge Process or sequence of accomplishment Generate relational database, add experience

13. RTTT 13 2/3/4 Mathematics Ontology © Regents of the University of California

14. RTTT 14 2/3/4 Algebra Sequence © Regents of the University of California

15. RTTT 15 2/4 Algebra Multiple Units © Regents of the University of California

16. RTTT 16 2/4 Algebra Part Unit © Regents of the University of California

17. RTTT 17 © Regents of the University of California RILS - 23

18. RTTT 18 2/3/4 Cognitive Readiness / 21 st Century Skills: Transfer static.flickr.com/80/230668852_055b631d8c_b.jpg © Regents of the University of California Content expertise Adaptive problem solving Situation awareness Decision making Self-management (meta-cognition) Teamwork Communication Resource access & management

19. RTTT 19 2/3/4 CRESST Cognitive Demands Content Understanding Adaptive Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning © Regents of the University of California Situational Awareness Risk Taking

20. RTTT 20 2/3/4 Problem Solving Understanding/ Comprehension Prior knowledge Domain knowledge Principles concepts Skills Abilities Schema Procedures Problem Characteristics Givens Parameters Goals Constraints Information attributes: -Credibility -Relevancy -Certainty -Coherence -Ambiguity Actions & Strategies Evaluate Seek Identify (Re)define Navigate Explore Plan Design Monitor Revise Solve Select Understand Solution Characteristics Types (single/multiple) Solution attributes: - Adequacy -Availability -Appropriateness

21. RTTT 21 2/3/4 Problem solving representation

22. RTTT 22 2/3/4 Key Elements of Assessment Design & Development Processes Vetted representations of knowledge and cognition-multi-purpose design Ontology anchors database, supports coherence Tasks based on models derived from research on learning, e.g., cognitive load, explanation, varied situations Authoring systems and reusable task elements Parsimonious, validated scoring guides exemplify expert performance Transfer task item bank

23. RTTT 23 2/4 Assessment Development Platform Collaborative space for users Training for teacher-based scoring, including discourse, models, criterion levels for raters Options for professional development for teachers and parents to assist in learning Student instructional options or units Expanding set of annotated examples of student work Alternative automated approaches to scoring Guidance and feedback for administration, including timing, requirements, privacy protection

24. RTTT 24 3/4 Summative Assessment Space Standards and representations structure database Collaborative space for designers Reviewed for linguistic features, instructional sensitivity, fairness, accuracy Data collection & analyses services

25. RTTT 25 3/4 Summative Assessment Space Meta-tags : representations/ situations/tasks/ student type/ unit/ sequences/criteria/ linguistics/ evidence by student Planned accretion/pruning cycles -coherence of interim and formative systems Validity prior to implementation Cost targets & monitoring Comparative benchmarks

26. RTTT 26 CA 7 th Grade Math Standards California

27. RTTT 27 CA MA AZ 7 th Grade Math Standards: Comparing AZ, MA, CA

28. RTTT 28 Multi-purpose Assessment Development Platform Architecture

29. RTTT 29 3/4 Summative Assessment Comments Technology-based systems afford integration of assessments used for interim or formative purposes in schools (with external) moderation to become part of a melded summative assessment Platform needs to permit integrated assessments following expert review, data, and report reviews Any summative assessment requires transfer tasks to avoid inappropriate teaching

30. RTTT 30 Summary: Transformation or Incremental Over the last 10 years, we have used tech- based measures interactively, over time, voice input, physical action sensors, collaborative Built semi-automated design systems on cognitive demands and ontologies, object- oriented, fully queried reporting systems, formative assessments with micro instructional interventions; some have been brought to scale of 10 to 30% the going price

31. RTTT 31 Summary: Urgent Evidence Needs New technical tactics to address extensive, adaptive performance,e.g., comparability Investment in small-sample statistical models to evaluate evidence before implementation Incentives for alternatives to current psychometrics with evidence of longitudinal growth, value-added, plus Achieved without compromising on the quality and fidelity of the measures and so that useful information is generated for teaching and learning

32. RTTT 32 Summary: Fairness & Cost Strategies for allowing student choice in assessment engagement New approaches to measuring classroom processes with technology Better privacy & security protections New procurement policies Please consider integrated assessment system

33. RTTT 33 http://www.cse.ucla.edu Eva L. Baker 310.206.1530 310.267.0152 baker@cse.ucla.edu voice fax email © Regents of the University of California