1. © 2008 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialPresentation_ID 1 A view on breaking barriers from a high tech industry John T. Behrens, Ph.D Kristen E. DiCerbo, Ph.D.

2. Presentation_ID 2 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential It takes a village to think about assessment (update) Dennis Frezzo, Cisco Bob Mislevy, UMD Philip Piety Patti West, Cisco Roy Levy, ASU AAron Crawford, ASU Daisy Rutstein, UMD Junhui Liu, UMD Youn Young Choi, UMD Dan Robinson, UT Mark Chen, Cisco

3. Presentation_ID 3 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Thanks also to  Bob Lissitz and organizers  State of Maryland

4. Presentation_ID 4 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential A play in 3 acts  Who we are and where we come from  Some barriers to be broken (with examples)  Some game play (or is it work?)

5. Presentation_ID 5 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Large and Global Diverse Students and Communities Students: Diverse Age, Gender, and Challenging Circumstances Communities: Mature and Developing Countries 165+ Countries 900,000+ Students/Year 3 Million+ Students since inception Cisco Networking Academy ICT Networking Skills Education for Individuals and Underserved Communities Diverse Educational Institutions Universities, Community Colleges, Vocational Schools, Secondary Schools, Non-profit Organizations, Second Chance

6. Presentation_ID 6 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Students by Region (900,000+ Students) 1% 7% 19% 2% 7% 5% 19% 7% 14% 18% % % of Global Total 20% 2% 7% 18% 16% U.S. and Canada Europe Latin America Central and Eastern Europe Russia and CIS 7% 5% 7% Middle East Africa Greater China APAC 1% Japan 17% Source: AME, jzinn_v1, Snapshot as of April 30, 2010 % of Worldwide Total as of July 31, 2010 

7. Presentation_ID 7 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Students by Education Level by Region (900,000+ Students) % of Global Total 20% 2% 7% 18% 16% U.S. and Canada Europe Latin America Central and Eastern Europe Russia and CIS 7% 5% 7% Middle East Africa Greater China APAC 1% Japan 17% Source: AME, jzinn_v1, Snapshot as of April 30, 2010 * See Notes Page for additional details on these definitions

8. Presentation_ID 8 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Product  14 courses  Entry-level tech skills  Instructor-led  Simulation and visualization software  Hands-on experience  Assessments Infrastructure  Educational process and learning systems  Larger server base than cisco.com  2.5 terabytes of data  1M assessments/month Relationships  Students  Academies  Instructors  Governments  Partners Curricula Program Design  Metrics  Services  Support Need to do more than make stuff

9. Presentation_ID 9 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Comprehensive blended e-learning 9 © 2009 Cisco Systems, Inc. All rights reserved.Cisco Confidential Curriculum  Explanatory text  Interactive media  Hands on labs  Embedded assessment  Rich feedback  Simulation supported Assessment  Knowledge focused questions  Student or instructor initiated  Integrated reporting  Rich feedback  Simulation supported Gaming  Promotes motivation and engagement  Promotes learning and practice  Provides larger context  Rich feedback  Simulation supported

10. Presentation_ID 10 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Summative Assessment SBAs, Final Exams, Midterms, Packet Tracer Formative Assessment, Measure Understanding Quizzes, Testlets, Feedback Activities Proficiency Flow Cert Practice Exams Instructor Initiated Student Initiated Proficiency Flow Chapter Exams, Practice Finals Flash Rich Media, Packet Tracer, Simulations, PT Practice SBAs Formative Assessment, Performance- Based Packet Tracer, Labs, Simulations Formative Assessment, Measure Understanding Formative Assessment, Performance- Based A comprehensive assessment model

11. Presentation_ID 11 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

12. Presentation_ID 12 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential BABY IT’S A WILD WORLD…..

13. Presentation_ID 13 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Audience Participation Pretend its 1990 (20 years ago) For each of the following tasks, consider how you would accomplish it (in 1990). 1.See your niece (who lives in another country) play with the present you sent her in real time. 2.Determine the height of Juday Creek in South Bend Indiana this morning. 3.Determine the amount of cloud cover in Australia today

14. Presentation_ID 14 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Audience Participation Jump back to the present (2010) For each of the following tasks, consider how you would accomplish it today 1.See your niece play with the present you sent in real time. 2.Determine the height of the Juday Creek in South Bend IN 3.Determine the amount of cloud cover in Perth, Australia today

15. Presentation_ID 15 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential What’s changed in 20 years? Remote sensors NOAA, Urban cams, Web cams Ubiquitous computing e-commerce, e-government Powerful computing (massively parallel computing – The Large Hadron Collider at CERN will produce roughly 15 petabytes (15 million gigabytes) of data annually – enough to fill more than 1.7 million dual-layer DVDs a year Computers for computers financial transaction; airline reservations

16. Presentation_ID 16 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential And you have it too Video and digital image sensors in your cell phone Social computing (vote for your favorite dancer with 1 million other people) Powerful computing given away for free Google apps, Google mash ups, Google earth Computers for computers Airline system that texts your cell phone Zotero Computers in space Tracking your location and giving you suggestions based on your location

17. Presentation_ID 17 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential The important aspect of the digital revolution  Is not about digitization, computation and information transfer (though keep buying the equipment)  It is about how our lives and work can be transformed by the technological changes

18. Presentation_ID 18 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential D e g r e e o f D i f f u s i o n o f t h e T e c h n o l o g i c a l R e v o l u t i o n Installation PeriodDeployment Period Big Bang Crash Institutional Recomposition Big Next Bang Intense Irruption funding of New Tech Between Frenzy Divorce paper& real assets Synergy Coherent Growth Maturity Idle money moving to new areas

19. Presentation_ID 19 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Assessment in the digital jungle Digital DesertDigital Jungle Periodic samples of dataOngoing ubiquitous data Small samples of dataDramatically large and ubiquitous Special intrusive systems to get dataData built into daily activity Lack of data requires special focused inputs “Items” no longer exist Absence of data requires inferential stretch Availability of data lessons need for inference Data scarcity leads to small sample science (e.g models of expertise) Data jungle leads to improved understanding of detailed mechanisms & rules (automated automated scoring) Disconnected intrusionsData storage leads to continuous improvement Data outside classroom not even considered Data is data no matter where it is

20. Presentation_ID 20 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Breaking boundaries in assessment  Hands on vs simulation  Curriculum vs assessment  Formative vs summative  Formal vs informal  Game vs assessment  Natural data vs tests  Results vs communication

21. Presentation_ID 21 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Assessment computing in the 20 th century Question  Answer(options)  Correctness  Points *b j

22. Presentation_ID 22 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential  Items as pieces  Value of independence  Scoring first

23. Presentation_ID 23 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential A More Flexible Idea If A and B then X1 = 1 If C then X2 = 0 X1 = 1; X2 = 0, Xn=? Select Activity Present & Record Evidence/ Feature Identification Evidence / Observable Synthesis Observables X1 X2 X3 X4 Xn Weights & Models Rules & Structure Activities Activity Meta-Data User Record Work Product If Z1 then Z2 -------------------- Max (f(x)) Task Level Feedback

24. Presentation_ID 24 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential In Sum Give Activity Start Score and Give Feedback Update Skill Profile Choose/Create Next Activity

25. Presentation_ID 25 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential The adaptive exam model Give Activity Start Score and Give Feedback Update Skill Profile Choose/Create Next Activity to optimize information

26. Presentation_ID 26 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential The practice tutor model Give Activity Start Score and Give Feedback Update Skill Profile Choose/Create Next Activity for best skill change

27. Presentation_ID 27 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential The motivational game model Give Activity Start Score and Give Feedback Update Skill Profile Choose/Create Next Activity best motivation

28. Presentation_ID 28 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Check it out….  The ECD delivery model is sufficiently flexible to show a high level symmetry between assessment, games and tutors.  The principles of inference and meaning generation from observation are articulated at a sufficiently broad manner to cut across the disciplines.

29. Presentation_ID 29 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential How we use ECD and technology to break assessment boundaries  Formal vs informal  Curriculum vs assessment  Hands-on vs hands-off  Education vs “real world”

30. Presentation_ID 30 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Packet Tracer cannot do everything a complex lab can do, but it can do many things a basic CCNA lab cannot:  Visualization  Scaling  Rapid revision  Off-line delivery  Complex scoring  Built in gaming  Micro-world authoring

31. Presentation_ID 31 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Exploration & Experimentation

32. Presentation_ID 32 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

33. Presentation_ID 33 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

34. Presentation_ID 34 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Complimentary Systems Filling in the missing pieces Fixed Response Questions Packet Tracer Based Assessments Focus Questions can pinpoint important knowledge and key ideas in specific parts of the curriculum. Students demonstrate real-world skills in simulated environment to “put all the pieces together”. Automated Scoring Question correctness easy to understand and discuss. Grade book supports transparency. New versions of Packet Tracer allow authoring of sophisticated scoring rules to make complex reports. Rich Feedback Multiple Choice Questions tied to personalized feedback with curriculum linking. PT SBA provide detailed diagnostic feedback regarding strengths and weaknesses in complex problems.

35. Presentation_ID 35 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Implication: Packet Tracer SBA Exam Launch 1 Take Exam 2 Submit It 3 Automatically Scored 4 View Feedback 5 35 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

36. Presentation_ID 36 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Packet Tracer SBA Detailed Feedback Helps Students Identify Areas for Improvement Proficiency EstimatesDetailed Scoring Feedback 36 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

37. Presentation_ID 37 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Simulation and feedback are threaded through the learning progression Course 1-3 Goal 1 Goal 2 Goal 3 Goal 4 Goal 5 Goal 6 Goal 7 Goal 8 Content Areas Chapter 1 PT Exam Quiz PT Integration Chapter 2 … PT Exam Quiz PT Integration …Chapter k PT PT Integration Quiz Exam SBA Labs Course 4 Labs Final Exam

38. Presentation_ID 38 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential COMPUTING FOR COMPUTERS But we are lazy and often stupid, so we want the computers to figure out the rules and the patterns.

39. Presentation_ID 39 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

40. Presentation_ID 40 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

41. Presentation_ID 41 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Hierarchical Cluster Analysis  We can see which “documents” are most like other “documents”

42. Presentation_ID 42 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Patterns of Command Sequences

43. Presentation_ID 43 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential CAN WE HAVE ASSESSMENT WITHOUT SUFFERING? But learning doesn’t happen only through instruction … And maybe learning can happen without suffering

44. Presentation_ID 44 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

45. Presentation_ID 45 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Roll the demo here………..

46. Presentation_ID 46 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Individual Performance V1C3 detail Work ProductMax Users1 FeaturesStatic IP Address on PC3192.168.0.99 PC3 is in Network1 Connectivity Tested1 ProficienciesBusiness Sense1 Configuration1 Troubleshooting1 Score300/300 V1C3

47. Presentation_ID 47 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Pulling it all together 47 © 2009 Cisco Systems, Inc. All rights reserved.Cisco Confidential Curriculum  Explanatory Text  Interactive media  Hands on Labs  Embedded Assessment  Rich Feedback  Simulation supported Assessment  Knowledge focused questions  Student or instructor initiated  Integrated reporting  Rich feedback  Simulation supported Gaming  Promotes motivation and engagement  Promotes learning and practice  Provides larger context  Rich feedback  Simulation supported Integration  Comprehensive skill model provides coherence  Common use of simulation supports cross-activity transfer  Possibilities for future research

48. Presentation_ID 48 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

49. Presentation_ID 49 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential And don’t forget to report it in a way that provides enablement

50. Presentation_ID 50 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential Conclusion  The digital revolution places us in the midst of incredible transformative social and economic change.  Simulation and digital gaming is embedded in this change.  Best practices in assessment may help us move forward in this area, the worst will not.  Large amounts of data and computing power will allow possibilities little imagined  We can only take advantage of these possibilities if we think comprehensively about ecosystems, design and meaning.

51. Presentation_ID 51 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential

52. Presentation_ID 52 © 2008 Cisco Systems, Inc. All rights reserved.Cisco Confidential