1. © 2003, Cisco Systems, Inc. All rights reserved.1 1 1

3. © 2003, Cisco Systems, Inc. All rights reserved.

4. MYTHS, FACTS, AND THE FUTURE OF NETWORKING ACADEMY ASSESSMENT
Ken Stanley

5. Thanks… The management of Cisco Systems, Inc.
The administrators of the Cisco Learning Institute
Our many partners in education

6. Assessment Issues and Solutions
Low test scores – partial scoring implemented for MCMA items
Readability – ESL and grade level
Miskeyed / trivial items – exam refreshes
Difficulty level - blueprinting exams and evidence centered design

7. Partial scoring – finals regrades are done
Initially - All or nothing
Now - Weighted model
Provides more scoring options for instructors.
All exams rescored except cert practice exam.

8. Readability
International reviewers – Exams for CCNA1 and CCNA4 have been posted for comments
CATCs have reviewed CCNA1 and CCNA4

9. Miskeyed, trivial, difficulty issues
Miskeyed items and very difficult items being removed
Very difficult items being removed
Item analysis are being completed
Hit list

10. SOME BACKGROUND ON ASSESSMENT IN THE NETWORKING ASSESSMENT
Presentation_ID
© 2003 Cisco Systems, Inc. All rights reserved. 10 10 10

11. The Program Continues to Grow, So We Have Some Decent Daily Volume

12. Myths Assessments are made by engineers
Cisco sets student passing scores
On-line exams should be the only source for grading
Certification exams are not related to the Academy
Tests are static for the life of a curriculum

13. Our Goal Is: Not just to make a lot of content
But make content that is good and usable by instructors

14. COMPLEXITY IN TASKS AND SCORING14 14 14

15. Illustration: NetPass Research Prototype
Create online performance assessment of networking skills
Focus on learner feedback rather than high-stakes testing
Learn something about assessment in general
You are invited

16. The Task Starts with a Scenario and Description of Goals

17. Diagramming Tool

18. The Diagram of the Network Is Created by Dragging and Dropping Icons

19. When More Detailed Information Is Required, It Is Collected as Well

20. When a Student Is Done, They Press “Submit”…

21. And the Graphical Representation Is Converted into a Text Representation

22. The Text File Is Scored Following Detailed Rules, which Result in Characterizations of the Work

23. And Feedback Is Created for the Student…

24. Virtuoso Was Designed with a Recognition of the Many Ways Assessments Are Used
Mastery learning
Repeated testing required
Multiple forms of exam desired
Formative assessment
Show questions and answers
Provide detailed feedback
Summative assessment
Provide higher level reporting
Keep item pools more secure

25. Compare Response to Answer Single Binary Number Comes out
Old Model
Old Model
Compare Response to Answer
Single Binary Number Comes out
Question Options
Limited to Text
Limited Right/Wrong

26. Task (Including Question)
New Model
New Model
Scoring
Output
Task (Including Question)
Could be any type
Scoring Rules Can Vary
Multiple Observables for Multiple Levels

27. Example 1: Show the Student Some Output…

28. Proficiency Reports/Personalized Feedback
Returning in version 3.1
Provides more than just percentage
Provides feedback to students

29. Attributes
Set by instructors
Number of attempts
Maximum activation window (more flexible login period)
Randomization of items
Duration of exam

30. COMPLETENESS: MAKING A COMPREHENSIVE ASSESSMENT SYSTEM30 30 30

31. Making a Comprehensive Ecosystems with the Certification Exam
The Academy Assessment team directs:
CCNA
CCNP
CCDA
CCDP exams

32. Thanks for attending Q AND A Session CNAP 09 32 32 32
© 2003 Cisco Systems, Inc. All rights reserved. 32 32 32

33. © 2003, Cisco Systems, Inc. All rights reserved.33 33 33