1. Using Vocabulary Services in Validation of Water Data May 2010 Simon Cox, JRC Jonathan Yu & David Ratcliffe, CSIRO

2. CSIRO. Talk outline Water Data Transfer Standards What are vocab services? How are they used? How are we are using it in WDTS? Future work

3. CSIRO. Joint effort: CSIRO & Bureau of Meteorology (BoM) Problem space: standardising format of water observation data BoM has mandate to integrate water quantity & quality data from 100s sources Providers use various software systems, with many different data export formats Water Data Transfer Standards (WDTS)

4. CSIRO. Water Data Transfer Format ( WDTF) Standardised XML format for transferring water related data (e.g. groundwater, river flow) Based on O&M v1.0 Primarily for water providers to send data for ingestion by BoM Also for exchange between other organisations Part of integrated national water information system to help with water (scarcity) crisis

5. CSIRO. Example: Reg200806.s3.9g Unclassified 7.56

6. CSIRO. Validating WDTF Over 200 data providers will submit WDTF Varying levels of technical ability Need automated conformance-testing WDTF validation service

7. CSIRO. Why not just XML Schema? XML Schema can’t capture some key business rules Reg200806.s3.9g Unclassified 7.56 Use Schematron to enforce co-constraints, complementing XSD Valid content and contextual use

8. CSIRO. Why not XML Schema? Content validation Reg200806.s3.9g Unclassified 7.56 Approp. identifiers

9. CSIRO. Content validation XSD can verify data-types and basic patterns … but many of the values are URIs Need to verify Valid URI i.e. verify the URI exists From a specific source i.e. check the domain name Co-constraints e.g. the uom is suitable for the property (expressed as a URI) Solution: couple XSD + Schematron validation service with a live ‘vocabulary service’

10. CSIRO. What are Vocabulary Services? Essentially interfaces to vocabularies Interfaces Web services: HTTP RESTful interfaces or SOAP Web application Java API Vocabularies Descriptions about a domain in specification language: concepts, properties, relationships, assertions E.g. water regulation codes, units of measure Example: HTTP API (interface) to SKOS/RDF resources (vocabularies)

11. CSIRO. Example of specification languages Limit to RDF/XML based RDF-based vocabulary languages: Simple Knowledge Organisation System (SKOS) Web Ontology Language (OWL)

12. CSIRO. Uses of vocabulary services 1.Dictionary lookup What does this term mean? What is ‘Metres’? 2.Discovery and analysis: What is this concept related to? What water regulation parameters use the unit Metres? 3.Interoperability and shared definitions and semantics: A parameter of WaterCourseLevel is measured in metres? Mine is in millimetres and there’s a conversion function available so I can interoperate – let’s talk 4.Data validation: Is this a valid water parameter? 5.Configuration management and code generation: Fill templates or Spit out some code or artifact based on concepts, properties, or conceptual structure in the vocabulary (sitemaps, Java code, Schematron rules)

13. CSIRO. Vocabulary Services in WDTS Leveraging Vocabulary Services for… Primarily for Data Validation Exposing set of codelists currently being maintained in Excel spreadsheet Represented in standard RDF format in SKOS SKOS allows re-use of vocabs Using SKOS gives the ability to define standard operations (using SPARQL queries) against vocabulary service for content validation E.g. validating URIs, contextual use of parameters and units of measures Key operations include: Does a concept exist? E.g. is this a valid unit of measure? Does a relationship between concepts exist? E.g. is the unit of measure used for measuring water levels valid?

14. CSIRO. Validation service architecture Vocabulary Service RDF Triple Store Validation Service User interface Schematron Rules XSD Validation HTTP Rest Interface Control Lists Transformation SKOS/RDF WDTF Vocab Interfaces SPARQL Queries WDTF Doc

15. CSIRO. Typical usage of vocabulary service in WDTS Specific vocabulary checking and cardinality constraints Using HTTP get queries like: Is this a valid vocabulary definition? http://localhost:8080/VocabLookup /get/concept /vocab1.0 /param%3AWaterpH_pH http://localhost:8080/VocabLookup /check/concept /vocab1.0 /param%3AWaterpH_pH

16. CSIRO. More query examples Is this Urn or http-Uri valid? E.g. urn:ogc:def:crs:EPSG::28349 http://localhost:8080/VocabLookup /check/property /vocab1.0 /dc:identifier /%27urn:ogc:def:crs:EPSG::28349%27 Does this Water regulation code parameter have the right measurement unit associated with it? http://localhost:8080/VocabLookup /check/relation /vocab1.0 /param:WaterpH_pH /skos:related /dc:identifier/%27[pH]%27

17. CSIRO. Conclusion WDTS Project and driving problem space for Vocabulary services Vocabulary services: Uses in validating content and business rules in WDTF

18. CSIRO. Future work A standardised SKOS Vocabulary Service Interface Need the right organisations to host common vocabs Who governs definition of UOMs, water regulations, etc? Continuing to develop validation solutions to current business rules for WDTF 1.0 Validation using complex business rules Further leverage of vocabulary services Documentation generation Leverage on vocabulary service to aid documentation generation – i.e. populating constraints Support for WaterML 2.0 Worldwide standard for a water data exchange format