Ensuring consistent spatial data management with FME Josh Goertzen Product Manager, IHS Tristan Wright GIS Technical Team Lead, IHS

We “acquired” FME back in 2005 Some background… We “acquired” FME back in 2005 Our development row: FME really came over to IHS with the acquisition of Ensight Information Services about 6 years ago. I was the only person who knew how to use it - I don’t think anyone else even knew what it was After spending some time as a floater where no one was sure where to put me - I eventually found my home with the data developers down a long hallway of individual offices. Traditionally these developers were using “c” code programs that they wrote to do things like generating spatial layers – change datums It was fundamentally a closed door system – just like our hallway most of the time I immediately began to be seen as a bit of a superhero each time I used an “off-the-shelf app” to get the same result much faster and with better consistency Slowly FME processes began to replace all others creating a unified platform for our spatial data management A quick story: We had a system set up to data enter some administrative polygon boundaries One of our gov’t sources stopped sending hardcopy maps and sent shapefiles instead and I was asked to help them figure out how to print it so they could enter it in

Our library has grown in the last few years… Some background… Our library has grown in the last few years… These changes allowed us to expand our spatial offering beyond what we had. Our library of spatial data wasn’t very large and most of the content was very simple outline layers used only as background wallpaper Over the last few years we’ve expanded that library immensely to include all different flavours of surface spatial data Detailed Hydro Recreation Watercourse Crossings Historic Sites Resource Management Zones Health Regions RCMP Districts Oil Sands Areas Detailed Oil and Gas Roads With the immensity now of our spatial offerings it makes it all the more important to have a proper spatial data management system in place Unfortunately, I’m now in a less-technical role but I’m very pleased to have been able to “pass the torch” to this next guy, Tristan Wright. Through his guidance I believe we have now migrated all of our spatial processes over to FME. He’ll walk us through the “meat” of this presentation

The FME advantage Integration of data from multiple data sources into seamless Canada-wide datasets Standardisation of data formats & structure Each commercial IHS spatial layer may have up to 15 individual source datasets that need to be standardised Reprojection & transformation of reference systems Data cleaning & verification Integration with other IHS data products to add additional content/value Individual update cycles for each layer (daily/monthly/annually, etc) Enable update of all or just specific data sources for a layer

Example: Data integration 5 data sources in different formats / coordinate systems 1 standardized commercial spatial layer

Example: Data cleaning (self intersections) Self intersection fixed using IHS custom transformer (a further extension of Safe Software sample code)

Fast data synchronization IHS relational database content updated daily. Spatial layers built based on the relational content. Need to be synchronized daily to capture content changes. “Complete” loads not possible in the processing timeframes available, FME allows fast synchronization of content using SQL and insert/update/delete database operations

Example: Data synchronization Additional content synchronized back to relational database SQL views added as readers to identify & process new/updated records Spatial layers synchronized using insert/update/delete operations

Adding further value to source data using FME Re-alignment of source data to standard IHS grid when supplied with DLS/NTS land descriptions Alignment of source data to roads/hydro and other IHS cultural layers Addition of new attributes/content based on existing IHS tables Example: adding lake/river names to hydro network polygons

Example: Hydro data value add Hydro data supplied with highly detailed geometry, but few river / lake names Reads source hydro data geometry Attempts to add missing names based on other existing IHS spatial layers (e.g. toponym layer) Writes new hydro spatial layers with extra river / lake name attributes

Automated data extracts Data extracts from our SDE database to GIS file formats (SHP/TAB/MID-MIF) Multiple projections for each province Custom extracts to IHS software packages in proprietary formats

Integration of other development tools into FME workbench Tool integration Integration of other development tools into FME workbench TCL Python Example: QA calling PL/SQL procedures for monitoring: Table counts Data Content using SQL queries Email log files

Example: Tool integration FME success / failure written to oracle log table PL/SQL QA procedures initiated from within shutdown script

The future… Improving automation Integration with automated file download processes (Kapow?) FME server Job management Scheduled tasks & automation Server based vs local machine based Central Repository of data processes Customer delivery of GIS files Data streaming to IHS software applications Further data cleaning (geometry errors)

Summary FME provides a platform for consistent spatial data management, and the flexibility to: read, synchronize and create commercial spatial layers based on multiple sources, data formats and coordinate systems Maintain individual data processes for over 300 spatial layers, each with different update frequencies. Add value to our source data by adjusting/refining source data, and incorporating additional content from existing layers. Integrate additional development tools for automating processes and monitoring/QA’ing processes

Thank You! Questions? For more information: Josh Goertzen (josh.goertzen@ihs.com) Tristan Wright (tristan.wright@ihs.com) www.ihs.com/energy