Principled Assessments of 21st Century Skills across Disciplines in a Community College Curriculum Louise Yarnall SRI International Jane Ostrander Foothill-DeAnza Community College District

Project brief National Science Foundation’s Advanced Technological Education program National Science Foundation’s Advanced Technological Education program Designing and testing instructional and assessment materials focused on 21st century workforce skills Designing and testing instructional and assessment materials focused on 21st century workforce skills Pedagogy represented a departure from pure technical content-focus Pedagogy represented a departure from pure technical content-focus

Definition of 21 st Century Skills Applying content knowledge and skills to solving real world problems Applying content knowledge and skills to solving real world problems More than a dozen Web-based workplace scenarios in different disciplinesMore than a dozen Web-based workplace scenarios in different disciplines Programming, network security, bioinformatics, engineering, environmental studiesProgramming, network security, bioinformatics, engineering, environmental studies

Assessment Design Linking content knowledge to soft skills Instructional context: multi-step, branching, self-paced Instructional context: multi-step, branching, self-paced Instructors: we’re not going to assess! Instructors: we’re not going to assess! We want them to assess; they’ll take the instruction more seriously! We want them to assess; they’ll take the instruction more seriously! Let’s make assessments: But how? Not just technical; it’s soft skills too Let’s make assessments: But how? Not just technical; it’s soft skills too Evidence-centered design Evidence-centered design Specifies knowledge to be measured and forms of evidence that demonstrate such knowledge (Pellegrino, Chudowsky, & Glaser, 2001)Specifies knowledge to be measured and forms of evidence that demonstrate such knowledge (Pellegrino, Chudowsky, & Glaser, 2001)

Assessment Design We documented our design decisions so instructors could do this too; co-designed with instructorsWe documented our design decisions so instructors could do this too; co-designed with instructors Form: Design patternsForm: Design patterns Not the assessments, but blueprints for developing assessments Not the assessments, but blueprints for developing assessments Design patterns have attributes that we specify and that can be re-used: knowledge and skills, work products, rubrics, task features Design patterns have attributes that we specify and that can be re-used: knowledge and skills, work products, rubrics, task features This approach gave us a chance to see common skills across disciplines—and differencesThis approach gave us a chance to see common skills across disciplines—and differences Principled Assessments for Design in Inquiry (PADI) (Mislevy & Haertel, 2006).Principled Assessments for Design in Inquiry (PADI) (Mislevy & Haertel, 2006).

Summary In this design pattern, a student is presented with a hypothetical user’s specific Web site development needs. Can students integrate the user’s needs into a practical software development process? Rationale It is important for students to have an opportunity to practice Web site development in a realistic, user-informed way because it helps students apply programming in the service of actual users. KSAs Skill of identifying and asking appropriate questions to specify user requirements Skill of identifying and asking appropriate questions to specify user requirements Skill of generating use cases based on user requirements Skill of generating use cases based on user requirements Skill of addressing user needs in specifying system requirements Skill of addressing user needs in specifying system requirements

Potential observations Identifying users’ primary needs for Web site use Identifying users’ primary needs for Web site use Define the user’s complete range of possible uses for the Web site Define the user’s complete range of possible uses for the Web site Identifying appropriate Web development tools Identifying appropriate Web development tools Potential work products Generate a list of written questions to ask of a user to identify and understand Web site needs Generate a list of written questions to ask of a user to identify and understand Web site needs Write a use case Write a use case Provide a written rationale for user interface decisions and selection of Web development tools Provide a written rationale for user interface decisions and selection of Web development tools Sketch or electronic mockup of user interface design Sketch or electronic mockup of user interface design

Lessons Learned Interdisciplinary insights Interdisciplinary insights Definitions of school learning outcomes Definitions of school learning outcomes

Interdisciplinary insights Assessment design for scenario-based curricula revealed commonalities among disciplines Research & analysis Research & analysis Problem framing & design constraints Problem framing & design constraints Generating a solution Generating a solution Applying tools Applying tools Communicating & working on teams Communicating & working on teams

Interdisciplinary insights And also revealed distinctions of… PrinciplesPrinciples Bioinformatics involves biology, engineering involves physics, and programming involves logic Bioinformatics involves biology, engineering involves physics, and programming involves logic ToolsTools Bioinformatics involves databases, engineering involves spreadsheets and experiment, and programming involves compilers and language Bioinformatics involves databases, engineering involves spreadsheets and experiment, and programming involves compilers and language

Definitions of learning outcomes 21st century skills involve learning: technical content knowledge and skills technical content knowledge and skills common ways of applying that knowledge in the real world common ways of applying that knowledge in the real world Engaging in knowledge application builds other, deeper forms of knowledge: blending content with application gives students a chance to develop mastery and expert knowledge blending content with application gives students a chance to develop mastery and expert knowledge schematic knowledge -- “knowing why” schematic knowledge -- “knowing why” strategic knowledge – “knowing when” strategic knowledge – “knowing when”

For more information (