1. Blue Coat and the Blue Coat logo are trademarks of Blue Coat Systems, Inc., and may be registered in certain jurisdictions. All other product or service names are the property of their respective owners. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Blue Coat Certification Exam Overview Education and Training Services May 2009

2. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Introduction  Prometric site  Configuring an exam  Structure of an individual item – Single response – TRUE/FALSE – Multiple response  Features – multiple choices, images, text decorations  Feedback and reports in Prometric  Goals of exam development 2

3. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Prometric Site  http://ibt.prometric.com http://ibt.prometric.com  Regular users (and customers) and admin users

4. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Editing Exam Configuration – Part A Exam result shown AFTER customer submits test and (optionally) writes feedback Unique identifier

5. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Editing Exam Configuration – Parts B and C For Beta-testing you may want to create “private” tests with keycode - No end-of-test feedback (to discourage cheating and complaining) - Display section scores - Do not mail results as we access them via Web Passing score – should be measured; currently it’s 70%.

6. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Exam Sections Failing a single section does not mean failing an exam! Results displayed after a BCCPP exam – the candidate’s overall score was 75% (fail). He complained - asked for a regrade. These are BCCPA sections: Total amount of items; each participant sees less than this

7. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Item Type: Single Response Unique identifier; prefix “AL” identifies chapter Section – may contain several chapters ProxySG items do not randomize PacketShaper tests typically do.

8. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Item Type: TRUE/FALSE HTML tags -,,

9. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Item Type: Multiple Response For some or all items we can remove random picking Better use chapter code for serial # (e.g. AB-017) Manually picked image User will get TWO warnings to select 3 answers More points == hard or important ?

10. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Individual Score Report – General Info Author Reports > Test Usage Individual Score Detail. Filter by exam (serial #) and date. Then pick the user by name or e-mail.

11. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Individual Score Report – Item Detail This report is useful to deal with complaining customers. Open problem: How much info do we disclose. (Currently we disclose the wrong items, wrong answer, correct answer and also analysis). “Complaining Customer Reports” are not hard to automate (but is not yet done). So we do them manually. Regrades happen often (more than half of complaining cases). Roberto can manually pass.

12. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Report: Detail Item Analysis To understand, what is happening with exam items, about 20-50 responses are necessary Not all response distributions are desirable, but here it is still hard to tell.

13. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only More Intelligent Exams?  Summary Item Analysis Report – responses to essays

14. © Blue Coat Systems, Inc. 2009. All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Other Reports  Author Reports > Custom Reporting – can get CSV (i.e. Excel) tables with most of the useful reports.  Test Statistics – histograms on how many people got any given result (can help to find the minimum passing grade)  Test Section Statistics – max/min/average/deviations for the results for each section in a given exam (could indicate problems with whole sections)  Summary Item Analysis Report – correct/incorrect answers for each item by serial #  Test Metrics – How many times each exam was requested  Item Timing Report – How much time was spent thinking about each item (can help estimate the total time needed for the exam)  Candidate/Group Improvement Report – comparing two exam events for candidate or group