NIEM Team, Oracle Public Sector Technology Introduction – September 2011 CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development XML Validation Test Suites with CAMV

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation The following is not intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Disclaimer Notice Slide 2

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Test Suites Introduction A key need in working with XML information exchanges, is developing examples and matching templates including rules, context, and roles for correctly managing information exchange content between partners. The W3C Schema by themselves are not sufficient. Learn how to use open source tooling, XPath rules, and the OASIS CAM templates standard to create automated testing tools. Integrating SQL table lookup validation rules The CAMV validation engine can also be used in production environments to manage live information exchanges. Slide 3

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Contents Part 1 The XML Validation Framework XPath Rules Techniques Code lists and SQL lookups Rules use cases Part 2 –Example CAM templates –CAMV validation examples –NIEM example test suite –Elections example test suite –SQL table database lookups –Running validations from visual CAM Editor –Results handling techniques –Summary * National Information Exchange Model (NIEM) – see Slide 4

XML Validation Framework Architecture and Deployment CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation XML Validation Framework CAMV XML instance Template (compiled) Database Lookup (optional) Results XML Java Handler Errors and Warnings StructureRulesCode Lists SQL Query Slide 6

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV Engine Slide 7

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV Validation Engine Standalone XML validator using XPath rules Context aware and declarative rules approach Integrated with CAM editor menus for instant testing External code list and SQL table lookup support Callable via Spring framework or Java API; thread safe and server container deployable Configurable error levels – error, warning Java error results object returned Embedded validation results attributes in XML results Slide 8

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM Editor / CAMV Environment CAM Editor CAMV XSLT Reports CAM template Single XML Test Instance XML Validation Result Slide 9

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV / ANT Script Environment XML Test Instances ANTANT CAMV XML Validation Results XSLT Reports CAM template(s) CAMV Control Script XML Slide 10

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV Test Control Script XML (single) Test-Template-v19.cxx Test-Instance-v19-3.xml true Param1 Param2 Validation Template Test XML instance Test case name Singleton Test Case Example Optional items Slide 11

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV Test Control Script XML (multi-instance)../samples/templates/EML-330-list-v7.cxx../samples/XMLsamples/UOCAVA/330 true Param1 Param2 Validation Template Test XML instances folder Optional items Test case name Multi-Test Folder Example Slide 12

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Linux GTK ANT Configuration Linux distributions (e.g. Ubuntu) has ANT pre- installed in /usr/share/ant/lib/usr/share/ant/lib Download Ant-contrib from Copy ant-contrib jar file to /lib directory inside ANT installation Slide 13

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Mac OS ANT Configuration Mac OS X 1.7+ “Lion” comes pre-installed - use command : ant –version and it will self-configure Mac OS X 1.6 and earlier – need to install “Xcode” development software first Download Ant-contrib from Copy ant-contrib jar file to /lib directory inside ANT installation Slide 14

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation ANT – Windows configuration Download ANT from Download Ant-contrib from Copy ant-contrib jar file to /lib directory inside ANT installation Set the ANT_HOME environment variable to the directory where you installed ANT Add %ANT_HOME%/bin folder to executable path Slide 15

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Windows System Path Settings Slide 16

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Running CAMV with ANT Go to CAMV location on Sourceforge Download CAMV release JAR and Test example ZIP Unpack test example from ZIP and put CAMV jar into folder Go to the command line (run cmd) Switch to folder where CAMV test suite is located ANT runAll Output is automatically placed in folder called CAMVTestResults located within the test instance location folder(s) Slide 17

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Reviewing Validation Results HTML Slide 18

XPath Rules Techniques Quick Syntax Primer Controlling Scope, Context, Evaluation Dynamic Structure Components Useful XPath Resources CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation XPath Syntax Primer XPath is a simple assertion language where expressions reference components hierarchically within an XML instance structure – e.g. /animals/canines/domestic/dogs/poodle[1] /animals/canines/domestic/* CAMV uses the XPath v2.0 Java library In CAM templates XPath is used extensively to provide rules and content details for XML structure components Slide 20

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation XPath Functions XPath Function samplerlogical concat(string,string,…) substring(string,start,len) string-length(string) normalize-space(string) upper-case(string) contains(string1,string2)Yes starts-with(string1,string2)Yes not(arg)Yes exists(item,item,...)Yes count((item,item,...)) /bookstore/book[price>35]/title Slide 21

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM XPath Extensions (most used) Structure FunctionsContent Functions makeOptional(xpath…)setNumberMask(xpath,arg) makeRepeatable(xpath…)setStringMask(xpath,arg) makeMandatory(xpath…)restrictValues(xpath,args) excludeTree(xpath…)setDateMask(xpath,arg) makeNillable(xpath…)datatype(xpath,arg) setChoice(xpath…)setLength(xpath,arg) setLimit(xpath…)lookup(xpath,args) setValue(xpath,arg) printMessage(xpath,arg)setNumberRange(xpath,args) setDefault (xpath,arg) Slide 22

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM templates and XPath expressions Within a CAM template XPath expressions are found in the section of the template. Slide 23

Rules Use Cases Typical Business Needs Quick Examples CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Standard Pattern for Business Rules Claim – An English statement of a business rule that conveys its meaning. Example: A Chapter MUST have a Title. Test – A formal statement of a business rule that can be parsed and processed by machine. Example: Diagnostic – An English statement that identifies and explains what happened or was found when a Test fails. Example: Chapter "X" does not have a Title. Outcome – Error, Warning, Pass Slide 25

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Controlling Scope and Context Condition controls if and when the content rule action applies Condition can control structure as well; optional, mandatory, exclude Condition can reference different component(s) for contextual rule actions Slide 26

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Some more examples Business Rule: Items with color attributes don't need a tray number—the color is sufficient for picking the item: Business Rule: If order weight exceeds 25 kg, purchase order must specify a freight carrier: 25" action="makeMandatory(//Item/FreightHandler)"> Using XPath axis referencing (when things are not adjacent in hierarchy): * Excerpted from Michael Sorens DEVX CAM article examples Slide 27

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Useful XPath Resources Michael Sorens article on using CAM from DEVX.com Martin Roberts – tutorial on writing CAM rules XPath tutorial resources Search on XPath tutorial Books available Tools – XPath expression constructors Slide 28

Code Lists and SQL Lookups Introduction to CAM code lists Advantages Tools and Hints CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Code lists CAM lookup() function allows referencing to external XML lists of values Can replace use of restrictValues() static lists Database - CAM lookup() function can reference SQL table for dynamic value checking Where to get code lists? Import XSD enumeration schema into CAM lookup lists Convert UBL Genericode into CAM lookup lists Dump from database into lookup list XML format Direct via SQL table lookup Slide 30

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM Code list XML format Textual value Code value Code list name Slide 31

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Combining Rules and Code lists Real power of code lists and CAM – ability to select code list values based on XPath rule criteria to handle different partner configurations, versioning and cross-table validations e.g. Ford / Mustang | Chevy / Corvette Message version id = 1.6 – use code list XYZ-1-6 Slide 32

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Code lists examples Load lookup list XML into CAM editor Assign lookup() rule to XML component Using SQL lookups – need to define connectivity to database and the SQL query to perform – we will do this part in the next section Slide 33

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Code Lists Summary Lookups can be to external XML file instances Simple XML format for optimized performance and readability Dynamic runtime control via XPath rules to select lists to be applied Lookup can reference SQL statement for database content checking Slide 34

Part 2 Example CAM templates Running CAMV validations Results handling techniques Summary CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM templates Here we introduce CAM templates and understanding their component sections Template is abstraction layer over schema structure and rules Template can be automatically built from existing XSD schema Add additional XPath validation rules Compiled templates provide the validation instructions for the CAMV validation engine Using CAMV validations and handling results Slide 36

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM Templates Overview Consists of 4 functional sections: Header declares properties and parameters for the CAM process to reference AssemblyStructure captures the business information exchange structures BusinessUseContext provides the rules to apply during validation Extension points to the annotations and external lookup tables Slide 37

Example CAM templates Examples Reviewed Quick walk through of important details CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Available Test Suite Examples Two test example templates and XML instances: Baseline SAR with just structure and content usage rules, schema parlance “cardinality, facets and enumerations” Baseline SAR advanced – logical business rules added to check consistency of information and structure components Each test set contains examples that pass and those that fail depending on the template validation applied * SAR – Suspicious Activity Report NIEM exchange Experimental SAR Examples OASIS EML V7 – UOCAVA Testing Set of test examples for Election Management: Both individual test cases and folder based multiple examples Each test uses different template and validation scenario * EML – Election Markup Language Standard Slide 39

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Experimental SAR Overview * SAR – Suspicious Activity Report Areas of Interest: -Drugs -Fraud -Gangs -Internet -Pornography -Property -Community -Weapons/Chemicals -Prescriptions -Travel/Immigration -Terrorism Slide 40

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Visual Template Rules Wizard Tools Slide 41

Running CAMV Validation Illustrative examples – pass, fail, warnings Default structure handling Data type and Content checking Cross-field validation rules SQL database lookup CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Example Advanced Rules Business Rule: Business Rule: Providing warning if PurgeDate is earlier than PurgeReviewDate Example of advanced rule Business Rule: Business Rule: Location must contain an address or a highway or coordinate set Slide 43

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Default Structure Handling By default all structure components are required and string content Content Masks Structure control Slide 44

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Running CAMV details Batch Command Line Execution WindowViewing Same Template in Visual Editor  From within the Test Suite folder on the command line execute > Ant runALL Slide 45

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Sample SAR Validation Results HTML Slide 46

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Installing and using SQL lookup example DB lookup sample install folder contents Contains test sample and template Run start and stop database scripts Run start and stop database scripts Simply download and unpack the ZIP archive and drop the DatasourceExample package into the samples Test Suite folder Then add the camv-test-DbList.xml to the testsuites folder Slide 47

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation SQL DB configuration details camv-test-DbList.xml contains data source reference and connection configuration Modify to match your local directory structure Modify inside file to match your local directory structure Modify inside dbProps.props file to match your local directory structure Slide 48

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Sample SQL DB Test Run Start the database instance (slide 47) From within the Test Suite folder on the command line execute > Ant runALL Slide 49

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation DB lookup test results rendered as HTML SQL DB lookup result Slide 50

CAM Editor Visual Tools SQL lookup entry wizard Run CAMV validation wizard Visual entry of SQL connection details Visual review of validation results CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM template SQL lookup rule definition This is the myLists.cam example template Slide 52

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAM template SQL lookup rule wizard Slide 53

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Test from CAM Editor validation Run Wizard jdbc:derby://localhost:1527/firstdb Pick test case XML file Choose JDBC driver Confirm driver name Replace default connection with actual JDBC string NOTE: this is the same information as contained in the dbProp.props file Slide 54

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Sample Visual Test Results SQL lookup result displayed Slide 55

More Results Handling XSLT script post-processing XML Diff considerations Regression Testing with CAMV Middleware Integration CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Default XSLT rendering of results HTML output Finds CAM validation results attributes Formats and outputs error details Slide 57

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Post-Processing Tips and Tricks XSLT post-processing not only for HTML reporting Create response error messaging directly from CAMV output Create receipt notification directly from CAMV output Can use simple DIFF to compare generic HTML report output to detect regression test changes Rule based checking is superior to simple result content change testing Java API and Spring supports direct integration of CAMV and returning validation results directly to middleware Slide 58

Summary Review Next Steps Future enhancements Resources and Links CAMV Test Model Data Deploy Requirements Build Exchange Generate Dictionary Exchange Development

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Review and Value Proposition Testing and verification of your exchange XML examples SQL database lookups and flexible code lists handling Business rule checking Post-processing options and reporting with XSLT scripts Regression testing with batch processing Flexible deployment of XML validation framework Integration via Spring API to middleware Slide 60

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation What are we working on? Improvements “CAM for web services” – currently exchange-centric Enhancing ANT capabilities and soapUI integration New features Dictionary integration with NIEM Domain Update Tool Evaluation tools for dictionaries and components Enhancing XML example generation with synthetic data Slide 61

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation CAMV Summary Open source, open public standards based, cross- platform deployment written using Java Uses Java APIs, Spring API, JDOM, etc. Validation features: XPath V2.0 support Looks ups 20 built-in functions Reporting Embedded CAMERROR, CAMWARN tags Spring API and results set handler Optional sysout test result log Slide 62

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation Links and References DOWNLOADS - #CAMV_Testing NIEM MATERIALS - Oracle NIEM website Oracle XMLOrb blog site Slide 63