1. EML Alchemy NIST Voting Data Formats Workshop Gaithersburg October, 2009 David RR Webber Technical Editor OASIS Election Services TC E-mail: drrwebber@acm.org

2. slide 2 Overview Change is constant (even in the conservative voting world) - how do we make trusted, cost effective information technology solutions, that improve democratic processes? How do we leverage and align standards based mechanisms and tools within our voting processes to make them more consistent, predictable, interoperable and verifiable? How does this fit into modern software engineering techniques, familiar best practices and popular development environments? Provide voting management staff the ability to vet and operate standard open processes without constant hand holding from technical staff.

3. slide 3 What makes something interoperable? Clearly defined open processes and exchanges Based on open standards and approach; not proprietary Adaptive approach with consistent methods This allows tailoring to specific local profiles and use patterns When requirements change can be adjusted consistently Fault tolerant and not brittle Ability to ignore non-critical interchange items and especially not to fail for trivial technical reasons not related to actual information details Leveraging XML capabilities to make self-describing transactions and exchanges possible Support new uses without extensive reprogramming Usage patterns set via external configuration allowing broad but controlled uses

4. slide 4 Agenda EML Technical Overview Introduction to transaction sets Typical simple illustrative use path OASIS interoperability event Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Massachusetts State Elections, IRV and EML Rapid EML localization tools Using OASIS CAM toolkit and EML templates Quick look – EML templates and NIEM IEPD process The Importance of Domain Dictionaries Summary / Review

5. EML Technical Overview Introduction to XML transaction sets Typical simple illustrative use path OASIS interoperability event

6. slide 6 The Voting Process Overview

7. slide 7 Why XML? XML is multi-faceted – it can represent: data (information exchanges) rules workflows and processes forms and reports configuration and control semantics (metadata) documents and contracts multimedia and presentations XML pervasive across software systems

8. slide 8 EML V6.0 Whats in the box? OASIS EML transactions – XSD Schemas EML XML dictionary – Excel spreadsheet Formal specification document EML templates – OASIS CAM templates* We will discuss EML transactions first… * CAM = Content Assembly Mechanism, OASIS standard template specification

9. slide 9 Category Overview of EML transactions Pre-election Election details (100) Candidates (200) Voters (300) Ballot Options (600) Election Voting (400) Post election Results (500) Analysis (500) Audit (350, 400) One or more series of XML schemas are provided to support each general process area.

10. slide 10 EML V6.0 Election Transactions Family Standard record structures to hold election information, ballot details, voting records and services Communicating specific result details on candidates and elections EML 520 – results reporting Results of election contest(s) and counts EML 510 - count Documents access to voting records and reason EML 480 – audit log Group of votes being transferred for counting EML 460 – votes group Actual record of vote castEML 440 – cast vote Used for voter authentication during a voting processEML 420 – voter authentication Used to register voters for an electionEML 310 – voter registration Details of actual voters for an electionEML 330 – voter election list Notification to voter of an election, their eligibility and how to voteEML 340 – polling information Describes the actual ballot to be used for an electionEML 410 – ballot Contest and candidates detailsEML 230 – candidate list Used to nominate candidates or parties, consenting or withdrawingEML 210 – candidate nomination Information about an election or set of elections.EML 110 – election event PurposeDocument Name EML 150 – election boundaries Boundary geo-districting information and polling places. EML 530 – statistics reportingStatistics relating to the election results and votes cast

11. slide 11 EML V6.0 – Supporting / Control transactions Used to manage security, audit control, access, system, setup and event tracking. List of accepted option proposals for referendum ballot formEML 630 – list of proposal options Confirm receipt of option proposal nomination detailsEML 620 – options receipt confirm Access and audit control to vote records, reason, eventEML 445 – retrieve vote Acknowledges receipt of vote from voting channelEML 450 – vote confirmation Voting token digital certificate channel use logEML 470 – voting token log Nomination of option proposals for referendumEML 610 – options nomination Voting channel device selection and useEML 360 – channel in/out detail General system level of messages for operational controlEML 350 – series of messages Confirm receipt and processing of EML transaction.EML 130 – response PurposeDocument Name EML 220 – response Candidate nomination response and confirmation.

12. slide 12 Example - EML 230 Candidate List OK-2007-09-1 Brad Henry John Wayne Bill Okapi Jane Smith Example of an EML artefact in XML

13. slide 13 Typical EML Process Path EML 230 candidate list EML 330 voter registration EML 410 ballot layout EML 440 cast votes EML 460 vote group EML 480 audit log activity EML 510 vote counts EML 520 election results Also consider EML 360 voting channel use EML 470 voting channel digital token log EML 530 election statistics reporting Pre-Election configuration Polling activity Tallying and Reporting

14. slide 14 OASIS Demonstration – London, 2007 EML 330 voter registration records Voter selects voting channel preference EML 440 retained by voting channels EML 460 sent to IBM tabulator system in Brussels Reconciled with EML 330 totals per channel EML 510 count results EML 520 voting results Illustrates the reconciliation of the number of authorised voters with votes cast and votes counted by channel and in total. White Paper available describing event: http://www.oasis-open.org/committees/download.php/25992/EML%20Interop%20Demo%20Report.pdf

15. slide 15 System interoperability testing Testing, Agreement / Alignment. Validate Templates Report System Uses & Creates Results Test Pass / Fail XML html 3 2 4 Test Cases XML Package Structure Rules Context Examples Documentation Dictionary Publish 1 Verify Result Outcomes XML Localization Content Hints

16. slide 16 Technology Considerations Summary Use of XML in software systems is pervasive All platforms and all major developer environments support XML Everything from FaceBook to Microsoft Office to Oracle database is XML capable The OASIS EML transaction Schemas can be directly used by developer software tools to handle the associated XML transactions The OASIS EML templates provide support for generating additional XML artifacts to suit localization needs

17. Recent US related EML work California Election Results Reporting Pew/Google VIP Project Geo-districting Use of EML in State Elections with IRV

18. slide 18 California Election Results Reporting Audience – news media channels Developed in summer of 2007 Enhanced EML 510/520 New EML 530 statistics transaction Joint effort between CA SoS staff and OASIS EML TC

19. slide 19 Outcomes Extend EML logical geographic constructs to handle USA district / ward structures Results for California 2006, 2007 and 2008 elections reported Validated ability of EML to report US election results Added extensible statistics reporting – same EML 530 now being used in Belgium as well

20. slide 20 Sample of California Election Statistics EML 530 XML EML 530 in web browser

21. slide 21 EML 150 - Pew / Google VIP Project Audience – voters District boundary, ballot and polling place information Mapping tool support EML 150

22. slide 22 EML 150 – Lessons Learned Re-usable components from EML allow rapid development of new transactions EML V6.0 dictionary provides core enabler EML 150 provides capabilities surpassing the original VIP XML layouts Includes GML standard mapping capabilities Polling place location, layout, access and facilities Aligned to OASIS EDXL for emergency management support OASIS EML technical committee able to rapidly respond to emerging needs and provide solutions VIP team is able to transition their existing data to new EML 150 format and gain benefits of broader information access and reuse (currently each state is new project).

23. slide 23 Massachusetts State Elections, IRV, EML Feasibility study on use of EML to support IRV for state-wide elections Determine whether EML would be an appropriate way to convey partially aggregated ballots from an IRV (or other preferential ranking system) from a lower jurisdictional level up to a higher jurisdictional level where the ballots would be counted and the result of the election would be determined Currently changes being considered for Massachusetts law to use IRV for state-wide offices OASIS EML TC members supporting investigation into using EML v6.0 Possible enhancements envisioned during EML v6.0 review period to precisely cover identified needs

24. Rapid EML Localization tools Using OASIS CAM toolkit and EML templates Quick look – Templates + NIEM IEPD process Importance of Domain Dictionaries

25. slide 25 Localization Key Components XML Schema 2 1 3 4 Documentation Readable information use details ( HTML ), EML cross-reference Spreadsheet XML artifacts The localized XSD Schema and picklist.xml Examples & Test Cases Realistic test XML instances for conformance and interoperability testing Key components (aka IEPD*) EML Schema XSD aligns voting information model, processes and domain dictionary *IEPD – Information Exchange Package Documentation

26. slide 26 National Information Exchange Model Support for NIEM approach Builds components for IEPD ( Information Exchange Package Documentation) Best practices - template evaluation for NDR compliance http://www.niem.gov OASIS at NIEM training event: http://events.oasis-open.org/home/ei-summit/2009

27. slide 27 NIEM and OASIS work OASIS Emergency Management TC - joint MoU between NIEM and OASIS Development of NIEM adaptor for handling OASIS XSD schema and domain dictionary definitions within NIEM Core Integrated into NIEM 2.1 (latest release) Successful formula / election services overlaps Opportunity to follow same path for OASIS EML localization for USA use Potential benefits: NIEM is Federal, State, Local and Tribal facing - developers already using NIEM Broad community resources for training/facilitation

28. slide 28 Localization Approach Steps EML XML XSD XML XSD Voting Exchange Manage develop 1 1 ingest 2 2 Base CAM Template Base CAM Template localize 3 3 Revised CAM Template Revised CAM Template Generate 4 4 Local XML XSD Local XML XSD Compare 5 5 EML Dictionary EML Dictionary Steps: 1.OASIS develops standard international XSD schema Implements the voting process, management and information exchange needs 2.Ingest the standard XSD into base CAM template 3.Apply localizations to CAM template Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed 4.Generate new localized XSD schema that conform to EML standard 5.Build dictionary cross-reference of localization usage 6.Publish documentation and XML examples Publish 6 6 Documentation + Examples Documentation + Examples

29. slide 29 EML V6.0 provides base CAM templates* Templates come pre-packaged in the EML V6 download * Open source Eclipse desktop tool: http://www.sourceforge.com/projects/camprocessorhttp://www.sourceforge.com/projects/camprocessor

30. slide 30 How it works in a nutshell EML xml EML xsd Localization Common xsd Namespace XML Schema Components CIQ xsd INGEST Structure Rules Documentation XSLT tools process entire EML XSD collection XSLT tools process entire EML XSD collection CAM = Content Assembly Mechanism OASIS EML XSD Collection CAM Template

31. slide 31 Example – Eclipse XML Structure Editor EML template is shown in the structure navigator panel for review and editing Template contains all the default content model and structure rules ingested from the XSD. All annotations and documentation from XSD also ingested (show as paperclip symbol). Code lists and typical content values inserted for easy visual reference.

32. slide 32 Using Editor to mark Exclude items Structure Editor Tools Select focus on item Invoke action menu (right mouse click) Pick action Tip: exclude rule display is context sensitive and only available on optional items Tip: use Add New Rule mode to specify different XPath for exclude (quick mode assumes current path)

33. slide 33 Extending the base EML schema In addition to excluding existing optional components you may need to add new pieces New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard Send comments and suggestions to election-comment@lists.oasis-open.org

34. slide 34 Adding New Domain Elements Select root element node in structure; right mouse click for context menu; select Add Namespace; then enter prefix and URI 1 Select element node in structure; right mouse click for context menu; select Add Child Attribute / Element; then enter prefix and name 2

35. slide 35 Generating your EML use schema Once your localization details are completed you are ready to complete the remainder of the process This starts with generating your own new subset schema Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation

36. slide 36 Generating subset use schema Select Export CAM as XSD menu option 1 File Menu Option Confirm the location and filename, and namespace mode. Select default for Mode to write a regular import set of XSD schema for each namespace 2

37. slide 37 Schema subset generated Set of XSD files with filename and namespace suffix Each namespace file is import for those specific type definitions Reviewing XSD results in a schema visual editor tool

38. slide 38 Information Restriction Considerations The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: setNumberRange(), setLength(), setLimit(), setDateMask(), setNumberMask(), restrictValues(), lookup() Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema Also option to set cross field validations using XPath conditional rules

39. slide 39 Dictionary and Documentation Tools Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible Documentation Tools Web interactive layout Business review tabular layout Developer raw template view Dictionary importance across process Discovery, reuse, consistency New development - library of components XML Tool support

40. slide 40 EML v6.0 spreadsheet dictionary

41. slide 41 Test Case Generator Feature Summary Make both Pass / Fail testing examples Content hinting so examples use real not fake data Test optional item logic with: all / random / none Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value) Make hundreds of test cases without manual editing Can link test case to XSD schema for structure tests You can modify XSLT tool for own testing needs

42. slide 42 Run CAM Rules Check on Examples Pick Test Case Example to VALIDATE; click Finish to run validation rules Review validation results Run Menu Option 1 3 2

43. slide 43 Localization Approach Recapitulation EML XML XSD XML XSD Voting Exchange Manage develop 1 1 ingest 2 2 Base CAM Template Base CAM Template localize 3 3 Revised CAM Template Revised CAM Template Generate 4 4 Local XML XSD Local XML XSD Compare 5 5 EML Dictionary EML Dictionary Steps: 1.OASIS develops standard international XSD schema Implements the voting process, management and information exchange needs 2.Ingest the standard XSD into base CAM template 3.Apply localizations to CAM template Exclude unwanted optional components Align local exchange needs to standard Extend for local needs, add new components as needed 4.Generate new localized XSD schema that conform to EML standard 5.Build dictionary cross-reference of localization usage 6.Publish documentation and XML examples Publish 6 6 Documentation + Examples Documentation + Examples

44. Summary Review and take-aways

45. slide 45 EML V6.0 Technical Highlights Set of standard transactions for core processes in elections Extended control, event and audit tracking transactions Domain dictionary of common components used in EML v6.0 OASIS CAM templates allow rapid localizations Set of IEPD artifacts generation Compatible with NIEM.gov approach

46. slide 46 Questions?

47. Localization Support Tools Quick Feature Review Sample Screenshots Tutorial Links

48. slide 48 Localization Support Tools Provides automated generation of need XML artifacts: Pick list development from template Schema subsets generation Dictionary cross-reference spreadsheet Use and rule documentation reports XML example instance and test cases Quick summary screenshots here Tutorial resources: http://oasis-open.org/committees/download.php/34489/EML%20and%20jCAM%20tutorial%20and%20primer.pdf

49. slide 49 Test Case Generator Feature Summary Make both Pass / Fail testing examples Content hinting so examples use real not fake data Test optional item logic with: all / random / none Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value) Make hundreds of test cases without manual editing Can link test case to XSD schema for structure tests You can modify XSLT tool for own testing needs

50. slide 50 Test Case Generation Quick Start File Menu Option Default directory to write examples into Name to be used for the examples How many examples to create Repeating elements count for Quick Test – just click OK to use default settings

51. slide 51 Test Case Results Active links to view the generated examples

52. slide 52 Run CAM Rules Check on Examples Pick Test Case Example to VALIDATE; click Finish to run validation rules Review validation results Run Menu Option 1 3 2

53. slide 53 Dictionary and Documentation Tools Dictionary Tools Create dictionary from CAM template Merge dictionaries together Compare to dictionary Excel spreadsheet compatible Documentation Tools Web interactive layout Business review tabular layout Developer raw template view

54. slide 54 Dictionary Compare Tool Select compare tool menu option Browse and choose EML dictionary

55. slide 55 Drag and Drop to Excel Spreadsheet Click OK

56. slide 56 Tabular Documentation Documentation Menu Option Select Format, resulting HTML opens in browser viewer

57. slide 57 Tabular HTML Content Details Enhanced Data type Logic Extended Code list handling Clear use pattern XPath references and functions Annotations

58. Resources Links to sites and examples

59. slide 59 IEPD - Information Exchange Package Documentation NIEM - National Information Exchange Model XSD - XML Schema Definition XSLT - XML style sheet language transformations Glossary

60. slide 60 Simple EML 440 Example Example using the EML 440 cast vote schema Simplified to minimum requirements Example can be downloaded here: http://markmail.org/message/qyrpfq75mtctblyq

61. slide 61 www.oasis-open.org/committees/election www.oasis-open.org/committees/cam www.oasis-open.org/committees/emergency www.niem.gov www.sourceforge.net/projects/camprocessor Resources: