© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Blue Coat Certification Exams Part 2: Creating Objectives, Items Education and Training Services May 2009

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Introduction  Requirements on exam content  The structure of exam data to be created  Developing objectives  Developing items  Versioning and releases 2

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Requirements  Exam content should be under version control (our SVN rather than Prometric) – Prometric Website allows accessing and editing anything, but it is difficult to ensure version history. Master copy should be on our server.  Exam content should be imported to Prometric from XML rather than entered manually – Avoid double effort and reduce human-caused errors  Exam content should be single-sourced ( ) – We create items in one place and then use them wherever it is appropriate.

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only High-Level Content Hierarchy: Sections  Subdivide the chapters into 3-8 large groups.  Give each group a short title (up to 25 characters) – it is a section  Decide, how many questions should be from each section  Ideally – formulate “3-8 most important things” covered by the given course. (You cannot expect the learner to remember, say, “20 most important things” – even if the course has 20 chapters).  Currently – we mostly do some ad-hoc grouping of chapters into somewhat related “clusters”. Also works fine

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Our courses and sections  svnroot/proxysg/trunk/v311/exams/conf/section- chapter-index-v321.xml

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Lower-Level Content Hierarchy: Objectives  Option 1: Exam measures the knowledge of the course book  Option 2: Exam measures the skills related to ProxySG (PacketShaper) that are deemed importand by SMEs

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only “General skills” and “specific skills” (or objectives)  Chapters are divided into “general skills” (denoted by A,B,C,... ) The expert opinion on skill importance could be used to assign weights to the exam items (not currently done). Skills could have different degrees of importance — they could range from very important (weight=4), to important (weight=3), to somewhat important (weight=2), to of little importance (weight=1)).  “General skills” are further divided into separate “specific skills” or objectives. Each objective is clearly stated by a verbal phrase (“Describe...”, “Identify...”, “Configure...”, etc.). Each objective is assigned to a specific slide in the textbook.

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only A Fragment of the “Objective Tree”  Refer to individual objectives as PROD.B.4 etc.  Traceability to things that do not change (as slide numbers sometimes do)

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Creating Items  After objectives (general and specific skills) are clear, can write items.  Prometric exam import requires specifically structured XML (accordingly to a DTD – Document Type Definition)  It does not matter, how we create this XML as long as it conforms to this structure.  Validating against DTD is not a guarantee that exam can be imported – some text fields may have limited length. (Prometric gives some unfriendly error messages in this case)

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only FrameMaker – XML Editor of Choice  Prometric allows import of XML, so we need to create or maintain data in XML format. – There is an alternative – create our own database application. More time-consuming (we would eventually creating our own mini-Prometric application).  Any good open-source XML editors?  FrameMaker is a strange XML editor, but it works – File > Preferences > General > Product Interface – can set this to “Structural Frame Maker” (allows editing any XML according to its XML Schema or DTD). – Use regular WYSIWYG FrameMaker (this is our approach).

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only The Way of FrameMaker  Item entry is in fact easy due to FrameMaker paragraph sequencing. Not following this order may cause invalid XML files. (A better XML editor? A Web form?)

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Template for a Single Response Item Stem Stepa Stepb Alternativea Alternativeb Answer Optional meta-info fields: Version, Hide, Tags May insert here an optional “Story”: (1) text paragraph, (2) code sample, (3) anchored frame with an image Points

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only What is Stepa/Stepb?  After the “Stem” (i.e. the formulated question for an item) there is an optional list of “Stepa” followed by one or more “Stepb”. Answer alternatives are something like “a & b only”, “a, b & c only”, etc.  This is our workaround to serve questions with multiple correct answers (emulated by questions with single response)

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Template for TRUE/FALSE Item  Same as “Single response” items; use only two alternatives – TRUE and FALSE

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Metainfo Fields (optional) All the fields not case sensitive; can omit.  Analysis (one or more paragraphs explaining, why the correct answer is correct)  ConformsTo (all, *-v311, v303, v311, v311-*)  Hide (No/yes) – “yes”, to exclude item from production and mock exams. Default is “no”.  Tags (comma separated list) – If tag “sample” is used, then the item is used for sample/mock exam. – Short strings that identify item’s topic. Items involving HTTP can have tag “http”. Anything related to byte cache can have tag “byte-cache”. Will use to search.

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Item Types Only three item types are recommended:  Single Response: Question stem has 2-5 answer alternatives; one of them is correct  TRUE/FALSE – Question should be answered as either TRUE or FALSE  Multiple Response (currently only in PacketShaper exams)

© Blue Coat Systems, Inc All Rights Reserved. Blue Coat Systems Confidential – Internal Use Only Assigning Points for an Item  Author assigns points for items. Possible strategies: – Currently used: All items are worth 1 point. In the limit it is normal distribution anyway. No partial credit. – More difficult items get more points (accordingly to previously observed statistics or beta-test results) – Harder-to-guess items in probability terms get more points. TRUE/FALSE is easier to guess correctly than to pick 3 from 5. (We do not play with this – we do not punish for guessing either) – More important skills get more points (This is recommended practice). Problem: We do not know yet, which skills are more important. – Any other strategy?