RTTT 1 Innovation and Technology in Assessment: Comments Eva L. Baker Director National Center for Research on Evaluation, Standards, & Student Testing UCLA Public and Expert Input Meeting, Race to the Top, U.S. Department of Education Boston, MA November 12-13, 2009 © Regents of the University of California

RTTT 2

RTTT 3 Overview: BIG challenges underlying innovative tasks Stimulate multi-level, multi-purpose system, open architecture, open source, scalable, top down- bottom up Invest in design not just administration Translate standards into rich representations of content, cognitive demands, situations and their relationships tuned for summative, interim, and formative uses Design assessments to support learning

RTTT 4 Big Challenges Obtain validity evidence, incentives for using new methods Exploit new options, games, immersive environments, simulations only if necessary for assessment information, anticipate change, SCORM Design to cost, with incentives for economies Save $ using authoring systems, reusable templates and objects for continued renewal & amortization of system Train users Find mid point between page-turning and dazzle

RTTT 5 1. Criteria for Innovative Tasks (other than efficiency) Emphasis on priority attributes of standard (s) in multi-step, realistic situations Clear expectations Teachable (or learnable), not trait-based Opportunity for multiple paths to solution or product Componential, reusable objects to permit quick creation of similar tasks Tagged to key dimensions in database Explicit feedback and areas for additional effort given

RTTT 6 1. Technology Features – off the shelf Hand-held for place-based learning, networking, collaboration,and teamwork, homework or informal assessment, feedback for teaching Sensors to permit mapping of movement, engagement in tasks, screen time Distraction, e.g., irrelevant animation Monitor student activity, feedback, revision processes to guide instruction Voice and handwritten inputs to broaden use Potential rrelevant inferences

RTTT 7 K Mapper Student Work

RTTT 8 1. K Mapper Student Action

RTTT 9 1. K Mapper Expert Scoring Model

RTTT Rescuing Animals Using Ratios and Equations

RTTT PuppetMan Math Problem-solver © Regents of the University of California

RTTT 12 2/3/4/ Focusing on first elements: essential design components Graphical representation of standards Content elements & relationships Cognitive demands Task situations Linguistic requirements Needed prior knowledge Process or sequence of accomplishment Generate relational database, add experience

RTTT 13 2/3/4 Mathematics Ontology © Regents of the University of California

RTTT 14 2/3/4 Algebra Sequence © Regents of the University of California

RTTT 15 2/4 Algebra Multiple Units © Regents of the University of California

RTTT 16 2/4 Algebra Part Unit © Regents of the University of California

RTTT 17 © Regents of the University of California RILS - 23

RTTT 18 2/3/4 Cognitive Readiness / 21 st Century Skills: Transfer static.flickr.com/80/ _055b631d8c_b.jpg © Regents of the University of California Content expertise Adaptive problem solving Situation awareness Decision making Self-management (meta-cognition) Teamwork Communication Resource access & management

RTTT 19 2/3/4 CRESST Cognitive Demands Content Understanding Adaptive Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning © Regents of the University of California Situational Awareness Risk Taking

RTTT 20 2/3/4 Problem Solving Understanding/ Comprehension Prior knowledge Domain knowledge Principles concepts Skills Abilities Schema Procedures Problem Characteristics Givens Parameters Goals Constraints Information attributes: -Credibility -Relevancy -Certainty -Coherence -Ambiguity Actions & Strategies Evaluate Seek Identify (Re)define Navigate Explore Plan Design Monitor Revise Solve Select Understand Solution Characteristics Types (single/multiple) Solution attributes: - Adequacy -Availability -Appropriateness

RTTT 21 2/3/4 Problem solving representation

RTTT 22 2/3/4 Key Elements of Assessment Design & Development Processes Vetted representations of knowledge and cognition-multi-purpose design Ontology anchors database, supports coherence Tasks based on models derived from research on learning, e.g., cognitive load, explanation, varied situations Authoring systems and reusable task elements Parsimonious, validated scoring guides exemplify expert performance Transfer task item bank

RTTT 23 2/4 Assessment Development Platform Collaborative space for users Training for teacher-based scoring, including discourse, models, criterion levels for raters Options for professional development for teachers and parents to assist in learning Student instructional options or units Expanding set of annotated examples of student work Alternative automated approaches to scoring Guidance and feedback for administration, including timing, requirements, privacy protection

RTTT 24 3/4 Summative Assessment Space Standards and representations structure database Collaborative space for designers Reviewed for linguistic features, instructional sensitivity, fairness, accuracy Data collection & analyses services

RTTT 25 3/4 Summative Assessment Space Meta-tags : representations/ situations/tasks/ student type/ unit/ sequences/criteria/ linguistics/ evidence by student Planned accretion/pruning cycles -coherence of interim and formative systems Validity prior to implementation Cost targets & monitoring Comparative benchmarks

RTTT 26 CA 7 th Grade Math Standards California

RTTT 27 CA MA AZ 7 th Grade Math Standards: Comparing AZ, MA, CA

RTTT 28 Multi-purpose Assessment Development Platform Architecture

RTTT 29 3/4 Summative Assessment Comments Technology-based systems afford integration of assessments used for interim or formative purposes in schools (with external) moderation to become part of a melded summative assessment Platform needs to permit integrated assessments following expert review, data, and report reviews Any summative assessment requires transfer tasks to avoid inappropriate teaching

RTTT 30 Summary: Transformation or Incremental Over the last 10 years, we have used tech- based measures interactively, over time, voice input, physical action sensors, collaborative Built semi-automated design systems on cognitive demands and ontologies, object- oriented, fully queried reporting systems, formative assessments with micro instructional interventions; some have been brought to scale of 10 to 30% the going price

RTTT 31 Summary: Urgent Evidence Needs New technical tactics to address extensive, adaptive performance,e.g., comparability Investment in small-sample statistical models to evaluate evidence before implementation Incentives for alternatives to current psychometrics with evidence of longitudinal growth, value-added, plus Achieved without compromising on the quality and fidelity of the measures and so that useful information is generated for teaching and learning

RTTT 32 Summary: Fairness & Cost Strategies for allowing student choice in assessment engagement New approaches to measuring classroom processes with technology Better privacy & security protections New procurement policies Please consider integrated assessment system

RTTT 33 Eva L. Baker voice fax © Regents of the University of California