© 2003, Cisco Systems, Inc. All rights reserved. 1 1 1

© 2003, Cisco Systems, Inc. All rights reserved.

MYTHS, FACTS, AND THE FUTURE OF NETWORKING ACADEMY ASSESSMENT Ken Stanley

Thanks… The management of Cisco Systems, Inc. The administrators of the Cisco Learning Institute Our many partners in education

Assessment Issues and Solutions Low test scores – partial scoring implemented for MCMA items Readability – ESL and grade level Miskeyed / trivial items – exam refreshes Difficulty level - blueprinting exams and evidence centered design

Partial scoring – finals regrades are done Initially - All or nothing Now - Weighted model Provides more scoring options for instructors. All exams rescored except cert practice exam.

Readability International reviewers – Exams for CCNA1 and CCNA4 have been posted for comments CATCs have reviewed CCNA1 and CCNA4

Miskeyed, trivial, difficulty issues Miskeyed items and very difficult items being removed Very difficult items being removed Item analysis are being completed Hit list

SOME BACKGROUND ON ASSESSMENT IN THE NETWORKING ASSESSMENT Presentation_ID © 2003 Cisco Systems, Inc. All rights reserved. 10 10 10

The Program Continues to Grow, So We Have Some Decent Daily Volume

Myths Assessments are made by engineers Cisco sets student passing scores On-line exams should be the only source for grading Certification exams are not related to the Academy Tests are static for the life of a curriculum

Our Goal Is: Not just to make a lot of content But make content that is good and usable by instructors

COMPLEXITY IN TASKS AND SCORING 14 14 14

Illustration: NetPass Research Prototype Create online performance assessment of networking skills Focus on learner feedback rather than high-stakes testing Learn something about assessment in general You are invited

The Task Starts with a Scenario and Description of Goals

Diagramming Tool

The Diagram of the Network Is Created by Dragging and Dropping Icons

When More Detailed Information Is Required, It Is Collected as Well

When a Student Is Done, They Press “Submit”…

And the Graphical Representation Is Converted into a Text Representation

The Text File Is Scored Following Detailed Rules, which Result in Characterizations of the Work

And Feedback Is Created for the Student…

Virtuoso Was Designed with a Recognition of the Many Ways Assessments Are Used Mastery learning Repeated testing required Multiple forms of exam desired Formative assessment Show questions and answers Provide detailed feedback Summative assessment Provide higher level reporting Keep item pools more secure

Compare Response to Answer Single Binary Number Comes out Old Model Old Model Compare Response to Answer Single Binary Number Comes out Question Options Limited to Text Limited Right/Wrong

Task (Including Question) New Model New Model Scoring Output Task (Including Question) Could be any type Scoring Rules Can Vary Multiple Observables for Multiple Levels

Example 1: Show the Student Some Output…

Proficiency Reports/Personalized Feedback Returning in version 3.1 Provides more than just percentage Provides feedback to students

Attributes Set by instructors Number of attempts Maximum activation window (more flexible login period) Randomization of items Duration of exam

COMPLETENESS: MAKING A COMPREHENSIVE ASSESSMENT SYSTEM 30 30 30

Making a Comprehensive Ecosystems with the Certification Exam The Academy Assessment team directs: CCNA CCNP CCDA CCDP exams

Thanks for attending Q AND A Session CNAP 09 32 32 32 © 2003 Cisco Systems, Inc. All rights reserved. 32 32 32

© 2003, Cisco Systems, Inc. All rights reserved. 33 33 33