1. 1 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 What Cisco’s ADI Group is Doing in Performance Testing PTC: June 18, 2004

2. 222 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Agenda Organizational Context Background of our Organization Assessment Strategy and Implementation Four Process Delivery Model Illustration: NetPASS Evaluation Moving Forward

3. 3 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Organizational Context

4. 444 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Areas we bring to bear: Statistical science Computing science Assessment science Psychological science Networking domain knowledge

5. 5 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Some background on our work in the Networking Academies

6. 666 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Global E-Learning Laboratory Over 450,000 Students Enrolled 30,000 to 40,000 Online Tests Daily Over 30 million Total Tests Taken 10,000 + Academies, 150 Countries Over 24,000 Instructors

7. 777 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 And there is probably one in your neighborhood….

8. 888 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 New Courses Sponsored by IT Industry Leaders:

9. 999 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 So we have some decent delivery volume…….

10. 10 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Assessment strategy and implementation in ADI

11. 11 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Some words about assessment implementation Assessment is about making inferences about knowledge, skills and abilities from limited data. Assessment development is not only about content, but about the models that allow us to make inferences from the observations.

12. 12 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Some over arching thoughts We conceptualize and work in assessment from end to end We have tremendous pressure to reduce time to market We will continually innovate in presentation, scoring and statistical technologies. All form assembly approaches require lots of items Items need to be complex to fight rote memorization This suggests assessment and task design and construction are strongest leverage points

13. 13 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Performance Tasks Need similarity to real-world tasks Cognitive Visual Physical Need to represent the relevant cognitive representations and features

14. 14 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Our primary framework for thinking about Assessment is Evidence Centered Design Provides a language for modern assessment Suggests structure for accomplishing the assessment Supports transfer and sharing

15. 15 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The world is complex ……..

16. 16 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 From Mislevy, Steinberg & Almond (in press): So is Evidence Centered Design How do we describe the world and the important claims and tasks in the world What components do the assessment tasks need? How do we score the responses from tasks? How do we pass that information to statistical analysis? How do we understand and report these inferences?

17. 17 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Performance Task Design The need and design of performance tasks is determined during the claims and evidence process What claims are we making about a student at the end of this course? (claims) What evidence will we look for that determines the degree to which a student has met that claim? (evidence) What situation can we set up for an examinee to demonstrate this evidence? (task specification)

18. 18 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Our goal is Not just to make a lot of content But to specify the underlying logic and design so many people can build off that specification Knowledge engineering of the domain This means we may need to track data at a more detailed level (e.g. field test items).

19. 19 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Claims database for CCNA

20. 20 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Contextualize the claim with scope specification and instructional guidance

21. 21 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Types of evidence you would look for

22. 22 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Representations to the student, and sources of variation

23. 23 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Representations from the student (work products), and features to detect

24. 24 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Add some additional information about relevant cognitive processes

25. 25 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Templates

26. 26 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Task Design Domain  A link to the claim or objective to which the task is seeking to provide evidence (e.g. Trouble shoot a 5 router topology running RIP with faults of type …..).  Enumeration of the relevant representations and artifacts (typology, routers, diagram, scenario) Features to vary in setting (# of routers, # of hosts, type of fault)  Relevant requirement dimensions (collect information, determine problem, apply fix, check, repeat) versus (apply fix to problem that is described).

27. 27 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Task Design Template – Census of Relevant Task Characteristics SettingCorporation Conference Center University Building LengthLess than 100m More than 100m Ethernet Standard10BaseT 100BaseT Subgroup NameTeacher Student Customer Bandwidth for a Subgroup Drop10Mbps 100Mbps Growth RequirementsGiven NA

28. 28 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Task Specification Template - Choosing Key Features SettingCorporation Conference Center University Building LengthLess than 100m More than 100m Ethernet Standard10BaseT 100BaseT Subgroup NameTeacher Student Customer Bandwidth for a Subgroup Drop10Mbps 100Mbps Growth RequirementsGiven NA

29. 29 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Use Selected Key Task Features to Make Display Model for the Scenario Problem Statement: 1.Setting sentence: A(n) setting is [create something that is a typical activity for this setting]. 2.Building size sentence: The setting is buildingLength long. 3.Network type sentence: The setting has been asked to install a Ethernet Standard network for this [the typical activity for this setting created above]. 4.subgroup 1 sentence: The subgroup 1 connections require a bandwidth of bandwidthForASubgroup 1. 5.subgroup 2 sentence: The subgroup 2 connections require a bandwidth of bandwidthForASubgroup 2. 6.subgroup 3 sentence: The subgroup 3 connections require a bandwidth of bandwidthForASubgroup 3. 7.Force closets sentence: No networking equipment can be stored in the subgroups 1, 2, & 3 area. 8.Location of POP sentence: The link to the internet is located locationOfExternalConnection(POP).

30. 30 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 3. Instantiate individual values into the template: Simple (LAN Design) A small university needs to set up a network for the English department. The English Building is 50 m long. The building needs a 100BaseT network for the English department. Each faculty member’s drop requires a bandwidth of 100Mbps while the student drops require a bandwidth of 10Mbps. The faculty and student areas are housed at opposite corners of the building. The LAN needs to have following additional characteristics: 1. The faculty area needs 8 network drops. 2. The student area needs 1 network drop. 3. One networking device should be placed in the student area. Use the device that would meet the minimum standards in the student area. 4. Students need access to 10 network connections to connect laptop computers. 5. No networking equipment can be stored in the faculty or student areas. 6. The link to the internet is located in the faculty area.

31. 31 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Advantages: A model that….. Guides task and test construction Provides extremely high efficiency and scalability Forms the basis of defensible evidence for high stakes tests By relating task characteristics to difficulty, we can create tasks with known properties in advance!

32. 32 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 And… These processes are being applied to simulation features as well

33. 33 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04

34. 34 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Mini-lecture: The Four Process Model for Assessment Delivery A language for modern assessment implementation

35. 35 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The Four Process Delivery Model Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

36. 36 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Activity selection determines the activities to be invoked in the presentation process Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

37. 37 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Evidence identification looks for important features of the work product (can be used for feedback). Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

38. 38 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Evidence identification looks for important features of the work product (can be used for feedback). Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

39. 39 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The result of evidence identification is values of observations Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

40. 40 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Evidence accumulation synthesizes the information for summary feedback Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

41. 41 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 In adaptive situations, evidence accumulation informs activity selection Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

42. 42 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Some other processes go on in the background Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

43. 43 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Four Process Model and Performance Tasks Performance tasks are really just a different means of presenting tasks to examinees

44. 44 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Evidence Identification of Performance Tasks Performance Tasks can be scored just like other more traditional tasks 0/1 (all or nothing) Partial credit (degrees of correctness) Different observables providing evidence for different student model variables

45. 45 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Evidence Accumulation and Performance Tasks Assessments containing performance tasks can still be summarized the same as more traditional assessments Add number of correct Ability estimates based on IRT Probability estimates based on Bayesian Inference Nets

46. 46 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Activity Selection and Performance Tasks Assessments containing performance tasks can still implement the same activity selection procedures as do the more traditional assessments Go to the next task in a specified sequence Randomly select a task from a specified bucket Task selected by examinee Task chosen through adaptive algorithm

47. 47 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Illustration: NetPass Research Prototype Create on-line performance assessment of networking skills Focus on learner feedback rather than high- stakes testing Learn something about assessment in general

48. 48 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The task starts with a scenario and description of goals

49. 49 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 To capture their mental model of the network, we ask them to draw the network with a diagramming tool

50. 50 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The diagram is created by dragging and dropping icons

51. 51 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 When more detailed information is required, it is collected as well

52. 52 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 When students are done, they press “Submit”…

53. 53 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 And the graphical representation is converted into a text representation

54. 54 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 The text file is scored following detailed rules, which result in characterizations of the work

55. 55 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 And feedback is created for the student…

56. 56 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 New Computational Tools Evolving all the time: A Bayesian Inference Network Design Network modeling TroubleshootImplement (Configure) Domain disciplinary knowledge Network Proficiency Media Addressing OSI Hardware Security Protocols IOS Correlation Part-of Prequisite Correlation OperationPlanning Part-of OSI

57. 57 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Summary Feedback – Probability of Mastery

58. 58 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Performance Tasks can also be Multiple-Choice

59. 59 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Assessment Evaluation

60. 60 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Exam Analyses

61. 61 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Examinee Comments

62. 62 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Item Performance

63. 63 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Moving Forward

64. 64 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Lessons Learned: Scoring Performance Tasks How can we get the most information from these performance tasks? Scoring the Product –What if there is more than one accepted outcome? Scoring the Process –More than one accepted process? –Is there really a ‘right’ way? Be Wary of Expert Judgment –Need to gather data to verify

65. 65 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 Future Directions Figuring out how to present the performance task is only one aspect to delivering a performance assessment The issues is really applying existing technologies and processes to more complex performance- based items and assessments

66. 66 © 2003, Cisco Systems, Inc. All rights reserved. Presentation_ID