SEEGRID Roadshow Technical Workshop Stuart Girvan – Geoscience Australia Rob Atkinson – Social Change Online CSIRO - Andy Dent, Simon Cox, Robert Woodcock

SEEGrid Roadshow 2005 Disclaimer Participants should only act on the information and advice in these presentations after independent consideration of their circumstances

SEEGrid Roadshow 2005 Program Demonstrator Architecture OGC's WFS Specification Introduction to Geoserver GML Application Schemas Implementing Geoserver - Database and Data Mapping Issues Use Cases and Service Structure for Clients in an Open Ended Architecture Further Technical Issues and Discussion

SEEGrid Roadshow 2005 Demonstrator Project Participants

SEEGrid Roadshow 2005 Contributors Rob Atkinson, Peter Barrs – Social Change Online Joan Esterle (UQ), Simon Cox, Rob Woodcock, Andy Dent – CSIRO EM Greg Jenkins, Mark Jolly, Terry Hannant – PIRSA Brendon Ward, Stephen Bandy – DOIR (GSWA) Lesley Wyborn, Aaron Sedgmen, Tim Mackey – GA Stuart Girvan – GA, pmd*CRC

Demonstrator Architecture

SEEGrid Roadshow 2005 Did everyone see the demonstrator this morning? Demonstrator

PIRSA DOIR GA Web Map Composer

Web Feature Services (OGC) GA XML - GML\XMML Web Map Composer PIRSA DOIR

PIRSA Web Feature Service (WFS) Common Interface Binding – GML/XMML GA Geochemistry Feature Data Source DOIR Geochemistry Feature Data Source DOIR Web Feature Service (WFS) GA Web Feature Service (WFS) Geoserver (Open Source) PostGIS (Open Source) Oracle PostGIS (Open Source) CLIENT APPLICATIONS DATA ACCESS SERVICES DATA SOURCES WebMap Composer GA Reports Application PIRSA Geochemistry Feature Data Source Little or no change required here Translation to standards here

Common Interface Binding – GML/XMML WebMap Composer GA Reports Application PIRSA WFS DOIR WFS GA WFS NTGS WFS MRT WFS NRM WFS NSWDPI WFS VICDPI WFS ? ? ? CLIENTS DATA SOURCES DATA SERVICES

SEEGrid Roadshow 2005 Services Registry Client Applications publish bind find Data Access Services Simple Web Services Architecture

SEEGrid Roadshow 2005 Client Applications bind For the moment not using registries to find or publish services Using “Magic Knowledge”, ie applications will know the data services available GA WFS GSWA WFS PIRSA WFS Services Simple Web Services Architecture

OGC's WFS Specification

SEEGrid Roadshow 2005 Web Feature Service (WFS) Specification (only 131 pages!) Uses http Accepts requests as either key-value pairs or XML Responds in XML (GML)

SEEGrid Roadshow 2005 Web Feature Service (WFS) Uses request/response not SOAP (although it can be wrapped in SOAP) Three main calls:  GetCapabilities  DescribeFeatureType  GetFeature

SEEGrid Roadshow 2005 Web Feature Service (WFS) Web Feature Service Get Capabilities Request Get Capabilities Response Describe Feature Type Request Describe Feature Type Response Get Feature Request Get Feature Response http protocol XML/ KVP XML XML/ KVP GML Schema XML/ KVP GML Data Source Config Files Application (web based or desktop)

SEEGrid Roadshow 2005 Features Features ≈ geographic features Features are Geography Markup Language (GML) encoded GML provides a standard for representing geometries (points, lines, polygons) and properties of geographic features

SEEGrid Roadshow 2005 Features Borehole  collar location  shape  collar diameter  length  operator  logs  related observations  … Fault  shape  surface trace  displacement  age  … Ore-body  commodity  deposit type  host formation  shape  resource estimate  … Observation  location  subject/specimen/station  property/theme/measurand  method  operator  date/time  result (+ type/reference system/scale/classification)  … Basin?  formations  shape – time dependent  resource estimate  …

SEEGrid Roadshow , Example: Instance of bridge feature encoded as GML GML attributes Features

SEEGrid Roadshow 2005 WFS Demo GetCapabilities DescribeFeatureType GetFeature

SEEGrid Roadshow 2005 Filter Encoding Specification Defines how to write WFS queries when making a GetFeature call

SEEGrid Roadshow 2005 Filter Encoding Example DEPTH 30 Geometry

SEEGrid Roadshow 2005 How would you use a WFS? Using http calls from your programming language to access the WFS and native parsers Using http calls and style sheets to transform the XML

SEEGrid Roadshow 2005 Eg GA Report Pages Java reads list of input URLs Subs in search values to pre-prepared XML chunks Calls URL of WFS(s) with XML parameters Parses XML document returned by WFS Data from XML document held in Java objects and sent to jsp pages for display

Introduction to Geoserver

SEEGrid Roadshow 2005 Why Geoserver? It’s the reference standard for implementation Open Source – we could influence and directly upgrade if necessary Open Source – no license fees/purchase cost

SEEGrid Roadshow 2005 Other WFS Products Deegree (open source), Cadcorp SIS Map Server, CARIS Spatial Fusion Enterprise, RedSpider Web 3.0, Map Manager, GenaWare, SclFeatureServer, JCarnacGIS, GeoMedia WebMap, MapXtreme (MapInfo), MapServer(UMN) more * Source: OGC Registered ProductsOGC Registered Products

SEEGrid Roadshow 2005 Geoserver Java based, servlets Runs using Tomcat (or embedded Jetty) Open source community size? Reliant on and closely connected to Geotools Lots of modules

SEEGrid Roadshow 2005 Setting up Geoserver Install into Tomcat using WAR file or use inbuilt Jetty servlet container Works with PostGIS, Oracle, ARCSDE, Shapefiles, mysql as data sources Config files catalog.xml, info.xml, schema.xml directory structurecatalog.xmlinfo.xml,schema.xml

GML Application Schemas

SEEGrid Roadshow 2005 Community/Application Schemas Community schemas provide the common or shared model The target community schema is essential

SEEGrid Roadshow 2005 GML Community Schemas Geological XMML - Exploration data (Example Geochemistry)Geochemistry ADX – Assay data GPML – plate tectonics XMML/GeoTime – stratigraphy CGI International Model Other LandGML MarineXML

SEEGrid Roadshow 2005 GML Application Schemas Design methodology can be learnt to develop community specific GML application schemas (Simon Cox, Andy Dent CSIRO) Equally important is who will be the custodian and manager of a community schema Also see GML 3.1 specification, Chapter 8 Rules for Application Schemes

SEEGrid Roadshow 2005 Informational Requirements Information models impact on real world performance Ontology Registries

SEEGrid Roadshow 2005 Why XML? Extensibility Self describing Ability to be (remotely) validated against schema XML Schema provides “loose tolerances” All software languages have tools to deal with XML

Implementing a WFS Using Geoserver – Database and Data Mapping Issues

SEEGrid Roadshow 2005 Traditional GIS and Standard WFS The WFS spec states the minimum requirements for a service interface It says nothing about private schemas vs public (community) schemas Hence most WFS software will only produce “flat” GML

SEEGrid Roadshow 2005 Flat GML Example Sample IDCoordinatesSample TypeSiO2%TiO2%Al2O3%Fe2O3%MgO% , -32Whole Rock , whole-rock

SEEGrid Roadshow 2005 Flat GML Example 50 1 Proc1 Blah 1 X Proc2 Blah 2 X2 0.01

Nested XML Example 50 SiO2 Proc1 Blah 1 X1 SiO TiO2 Proc2 Blah 2 X2 TiO2 0.01

SEEGrid Roadshow 2005 XML Structures There are potentially many structures or information views that may be appropriate for describing data These may be influenced by use cases and architectural considerations

SEEGrid Roadshow 2005 Data source to community schemas Community schemas provide the common or shared model All data providers have their own local data model  All data providers must map data from local source (database) to community schema, irrespective of technology implementation

SEEGrid Roadshow 2005 Requirement We need to be able to map from private schemas to public in as flexible a manner as possible

SEEGrid Roadshow 2005 WFS Spec Problem WFS spec allows any query based on the feature description (community schema) Comes from the point of view that you’d be able to use, for example, xpath against a ‘notional’ XML document In reality almost all data sources will be databases or proprietary formats of some kind In practise want to limit the potential queries that can be made against the data source

SEEGrid Roadshow 2005 Query Profiles Why you should only allow certain queries  efficiency  security  user friendliness  legal requirements Eg phone books, amazon.com

SEEGrid Roadshow 2005 Requirement Limit the possible queries that can be made against the feature data source

SEEGrid Roadshow 2005 Implementation At beginning of project Geoserver (nor any other software we were aware of) could not perform mapping from private database to community schema Geoserver could only serve “flat” GML No restrictions on queries (follows the WFS spec)

SEEGrid Roadshow 2005 Geoserver Enhancements Initial stages of project included upgrading Geoserver to include plug in mapping to complex schemas and ability to configure pre-defined queries Info.xml and Schema.xml files Catch – only works from database source that you can run an SQL query against

SEEGrid Roadshow 2005 info.xml

SEEGrid Roadshow 2005 schema.xml

SEEGrid Roadshow 2005 Mapping & Query Limitations Limited amount of branching Restricted in the positioning of branching No data dependent tags Only one predefined query per feature type

SEEGrid Roadshow 2005 Coping with Limitations The impacts of the limitations of Geoserver and WFS were accommodated by  agreeing to slightly modified target schemas  agreeing to commonly defined input parameters and patterns  a small number of technical hacks

SEEGrid Roadshow 2005 Geoserver Development The Geotools community is considering a comprehensive solution to the problems of complex objects. See Types Types If you’re very keen get involved

SEEGrid Roadshow 2005 Technical Lessons

SEEGrid Roadshow 2005 Baseline Software Requirements* Compliant with current WFS specification Separation of back end system from common interface Ability to restrict incoming queries *For equivalent of current project

SEEGrid Roadshow 2005 Baseline Software Requirements Ability to access back end data source of choice (potentially through firewall) Ability to produce GML documents with at least single branching complexity, element attributes, and nestable single elements

SEEGrid Roadshow 2005 Software Wishlist Ability to map to a GML document of any complexity and a GUI interface to configure the mapping Ability to run various queries against a single defined feature type Ability to map to an element based on a data source value

SEEGrid Roadshow 2005 WFS Spec Changes As a result of the project there is a formal change request in with OGC to allow for query profiles for a feature This will also require a minor change in the Filter Encoding Specification

SEEGrid Roadshow 2005 Moving forward The SEEGRID Roadshow will extend the test bed to the remaining state and territory surveys for Geochemistry Considering a 3 year ICIP (Industry Co-operative Innovation Program) grant which will take it from a test bed to a production system (network)  The Industry in ICIP is critical

SEEGrid Roadshow 2005 Moving forward The establishment of a production level network of services for one data type will greatly smooth the way for other types of info, not just Geological Opportunities to cooperate with SEEGRID to progress technological and standards solutions  As part of ICIP proposal or independently SEEGRID is free, contributions to common goals come from sub sets of the community

Use Cases and Service Structure for Clients in an Open Ended Architecture

User wants to select sample data by some bounding box  Sample data must be queryable by bounding box Scenario 1

User wants to see details of data selected in bounding box  Sample data must be queryable by bounding box or by id numbers of selected records Scenario 2

User wants to select sample data by some bounding box and some threshold value  Sample data must be queryable by bounding box and threshold value property Scenario 3

SEEGrid Roadshow 2005 Client Requirements Availability, SLA’s Support Response times Data quality, metadata

SEEGrid Roadshow 2005 Server Considerations Not all use cases/query profiles can be known But you can probably make a reasonable guess as to what may be required Most importantly, like any project, need to involve clients (if possible)

SEEGrid Roadshow 2005 Client Strategies for Using WFS Querying everything vs multiple queries Schema dependency Caching vs going back to the web service

Further Technical Issues and Discussion

SEEGrid Roadshow 2005 Technical Lessons MCAProjectOutputs/TechnicalLearningfromtheMCA.pdf

SEEGrid Roadshow 2005 WFS in the Notional Architecture Notional Architecture is the end game perfect world A brief note – what are registries about? Imagine you’re an application!

Features Coverages Models Real-time data Data Access Services Ontology Registry Service Metadata Client Applications Persistent Bindings Feature Type Catalog Model Management Services publish bind find Dataset Metadata Processing Services Notional Architecture

Notional architecture Dictionary Service Metadata Persistent Bindings Feature Type Catalog Dataset Metadata Yellow Pages + Semantics MANAGED Features Coverages Models Real-time data Data Access Services Registry Client Applications Model Management Services publish bind find Processing Services Managed Dataset Metadata Someone has vouched that from this service you can get a sparkplug that has a certain pedigree/quality Metadata - it was made in Germany by Mercedes Benz Managed Persistent Bindings Someone has created instructions to use this service in the following ways Persistent Binding (ie always true) - only ever put the sparkplug the right way up For a Commodore you need sparkplug X Managed Service Metadata Someone has vouched that a spare parts dealer service exists that supplies sparkplugs Service Metadata – Add in the yellow pages for the spare parts dealer Including details of how to contact (use) that service Managed Feature Type Catalogue Feature Type Catalogue – List of Feature Types eg Spark Plugs, Petrol Caps, Oil Filters Feature Type - The agreed set of attributes a sparkplug must have Thread Diameter, Thread Reach, Gap etc