Daniel Pilon Senior project officer at NRCan Using FME for Topographical Data Generalization at Natural Resources Canada Daniel Pilon Senior project officer at NRCan

Overview Map generalization at NRCan Principles of good map generalization ETL generalization patterns

Very complex: manually or automatically Map Generalization The process of reducing detail on a map Very complex: manually or automatically

Generalization Paradigm Shift Manually Comply to very strict specifications Prepackage products 250K, 1M, 7.5M Finality on its own Art Craft Process Automatically (faster turnaround) Less complex specifications On demand scale production Integrated into other products Commodity Process

Generalization Paradigm Shift Example WMS Web Mapping Service 1 : 30 000 000 1 : 7 500 000 1 : 1 000 000 1 : 250 000 1 : 50 000

Generalization Paradigm Shift Example WMS 1 : 40 000 000

Generalization Paradigm Shift Example WMS 1 : 5 000 000

Generalization Paradigm Shift Example WMS 1 : 1 000 000

Generalization Paradigm Shift Example WMS Threshold: 1: 120 000 1 : 119 000 (serving 1: 50 000) 1 : 121 000 (serving 1: 250 000)

FME for Generalization…Why? Few generalization solutions No complete generalization solution Some European solutions Very complex software to use… Tackle European problems

FME for Generalization…Why?

FME for Generalization…Why? Few generalization solutions No complete generalization solution Some European solutions Very complex software to use… Tackle European problems FME as a generalization tool… General purpose spatial manipulation engine Reliable and fast Easily add extensions (python)

Overview Map generalization at NRCan Principles of good map generalization ETL generalization patterns

The principles of Good Genralization Automation Generalization cases Simple and efficient decomposition of the tasks performed by the cartographers Objective tools to characterize map objects Tools to edit modify the spatial objects Reactive control on the map object’s state during the process Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

Generalization Case It is a formalization of the cartographer’s knowledge (generalization rules) and a communication tool between the cartographer and the software engineer Generalization cases do not tell us what algorithms or what parameters to use

Generalization Case example For the Vegetation For all the Vegetation which breaks the constraint Dimension: Minimum area If the Vegetation breaks the constraint: Position proximity Aggregate the area Else Eliminate the area

The principles of Good Genralization Automation Generalization cases Simple and efficient decomposition of the tasks performed by the cartographers Objective tools to characterize map objects Tools to edit modify the spatial objects Reactive control on the map object’s state during the process Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

Measures are used: Measures To calculate the characteristics of map objects (or group of map objects ) Before the generalization: to know how to generalize After the generalization: to assess the success of the applied generalization operations

Measure Examples Unary measures Length Area/Perimeter Circularity ratio Oriented Bounding box Triangulation Bounding box

Measure Examples Binary measures Spatial relationship Distance Triangulation

Measure Examples N-ary mesaures H3 H1 H2 S1 H1 H1 H3 H1 H1 H3 H2 H2 H2 Stream ordering (Strahler or Horton)

The principles of Good Genralization Automation Generalization cases Simple and efficient decomposition of the tasks performed by the cartographers Objective tools to characterize map objects Tools to edit the spatial objects Reactive control on the map object’s state during the process Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

Generalization Operators Generalization operators are typical transformations applied on spatial objects. They allow the decomposition of the generalization process into several sub-problems in order to manage complexity Generalization operator are implemented using different generalization algorithms. Ex.: For line simplification: Douglas & Peucker, Lang, Sherbend

Generalization Operator Examples Selection Simplification Smoothing Collapse

Generalization Operators Examples Aggregation Typification Displacement

The principles of Good Genralization Automation Generalization cases Simple and efficient decomposition of the tasks performed by the cartographers Objective tools to characterize map objects Tools to edit modify the spatial objects Reactive control on the map object’s state during the process Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

Constraints Constraints are rules applied to data in order to comply with requirements of the target map specifications (ex: An area must respect the minimum size threshold) Constraint are used: To trigger a generalization operation To assess the result of a generalization operation To achieve generalization, a feature must fulfill several constraints

The principles of Good Genralization Automation Generalization cases Simple and efficient decomposition of the tasks performed by the cartographers Objective tools to characterize map objects Tools to edit modify the spatial objects Reactive control on the map object’s state during the process Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

Generalization Behaviour Provides the link between constraints, generalization operator and measures Implements mechanism for choosing algorithms and parameters Implements mechanism for choosing alternate scenarios in case of constraint failure

Generalization Behaviour Example for Line Simplification ETL Generalization Patterns Reset to the original geometry Determine an alternate algorithm and/or parameters Failed Measures on the feature Apply the algorithm on the feature Asses the results through a set of constraints Determine the algorithm to use Success

Overview Map generalization at NRCan Principles of good map generalization ETL generalization patterns

ETL Generalization Patterns General and reusable solution to a commonly occurring problem in generalization when used in an ETL context Three stages pattern Meta-algorithm pattern

Three Stages Pattern Used when generalization operator(s) are applied on different part of the same feature Solution: Create a pseudo-object representing each generalization operation to be done on the real objects Evaluate the constraint against the pseudo-objects created in 1 Apply on the real objects the pseudo-objects that meet the constraint evaluated in 2

Three Stages Pattern Example Example with amalgamator operator If you implement a one stage process

Three Stages Pattern Example Example with amalgamator operator If you implement a three stages process First create the amalgamation zones Second you remove unwanted zone Dissolve the area

Meta-Algorithm Pattern Used when it is too complex to program the behaviour and the constraint in an ETL Solution: Select the algorithm(s) Select the constraint(s) to implement Select a behaviour pattern Implement the solution in a high level language Wrap the solution in a transformer

Meta-Algorithm Pattern Example Example with Sherbend Implementation of the Wang algorithm

Meta-Algorithm Pattern Example Line simplification Basic generalization operation We all thought of the Douglas-Peucker algorithm to solve that problem… Does it simulate the work of the cartographer well enough? An example with contours

Meta-Algorithm Pattern Example Contour line simplification Red: Original contours Black: Contours with DP 50m

Meta-Algorithm Pattern Example Contour line simplification Black: Contours with DP 50m The results are unacceptable Does it simulate the work of the cartographer? No Douglas-Peucker line simplification is not doing a good job of generalizing natural features

Meta-Algorithm Pattern Example Contour line simplification Red: Original contours Black: Contours with Sherbend algorithm with a parameter of 150m The Sherbend algorithm will remove the bends below the tolerance and keep the ones above it

Meta-Algorithm Pattern Example Example with Sherbend Implementation of the Wang algorithm Implementation of constraints Self intersection Line crossing Sidedness

Meta-Algorithm Pattern Example Implementation of constraints Self intersection Line crossing Sidedness

Meta-Algorithm Pattern Example Example with Sherbend Implementation of the Wang algorithm Implementation of constraints Self intersection Line crossing Sidedness Implementation of a scenario to resolve conflicts (behaviour pattern)

Meta-Algorithm Pattern Example Scenario to resolve conflitcs Implementation of an iterative process to resolve constraints

Meta-Algorithm Pattern Example Example with Sherbend Implementation of the Wang algorithm Implementation of constraints Self intersection Line crossing Sidedness Implementation of a scenario to resolve conflicts (behaviour pattern) Creation of a new transformer

Meta-Alorithm Pattern Meta-Algrorithms can be developed by user in FME FME Python extension (Python caller) Open source libraries Shapely for spatial manipulation and spatial relationship (http://pypi.python.org/pypi/Shapely) RTree for spatial indexing (http://pypi.python.org/pypi/Rtree/) Already implemented the following meta-algorithms Douglas Peucker Convex Spike Smoothing Sherbend

Future Works Develop new generalization patterns applied to ETL Develop new meta-algorithms Start the generalization of the hydrographic network

Thank You! Questions? For more information: Daniel Pilon Natural Resources Canada dpilon@nrcan.gc.ca WWW.GEOGRATIS.GC.CA WWW.GEOBASE.CA