Salvador Bayarri Consultant, World Bank

 The SDI architecture model  Software components: the SDI stack  Basic services  Metadata  SDI access control  Current trends in SDI technology

 SDI uses a three-tier service-oriented model Source: mapit.biz Standardized

Same thing with a component-oriented view: Clients Middleware Servers FeaturesCoverages Metadata update Catalogs Geo-processing and catalog Services Content Repositories Other data e.g., administrative, statistical, env. reporting Service chaining: search, display, access, e-commerce, …. Direct data access Access to transformed data, pictures, maps, reports, multi-media content Metadata search and retrieval for data and servicesGIS

 The use of a communication layer based on standardized Web services ensures: ◦ Transparent access to data, regardless of format, technology… but we still need to harmonize the data content ◦ Interoperability between different brands of servers and clients ◦ Distributability of components in a network. The SDI philosophy is that each responsible organization maintains and publishes its datasets. Multiple clients, catalogs, etc. are possible

 Spatial DB = Relational DB + … ◦ Geometry representation. Standard is SFS (Simple Feature Specification) ◦ Spatial Index. Speeds up search and operators, sorting objects by location ◦ Spatial operators. Can be invoked in SQL sentences to measure, perform geometric operations (e.g. intersection, reprojection), create new objects, etc. Essential for data quality assurance.  Needed for large datasets, many simultaneous accesses (e.g. concurrent editing)  Relational databases also used for metadata  Files still used for raster data (images, elevation grids)

 Example of spatial table Encoded geometry field

 Simple Feature Specification (OGC) for SQL SDI stack: Spatial Databases

 Well-Known Binary representation for simple features (OGC)

 Open-source (OS) implementations ◦ SpatiaLite (extension to sqLite), supported by qGIS ◦ MySQL spatial extensions (not fully standard) ◦ PostGIS (extension to PostgreSQL)  Excellent support for standards  Widely supported in OS GIS, ArcGIS, Intergraph…  Used by GeoNode  Friendly administration tools (pgAdmin)  Interesting extensions (pgRoute for routing) The whole UK Ordnance Survey is stored in PostGIS (£1,000,000 savings compared to Oracle Spatial), as well as the national topographic data of France for the Institut Géographique National

 Proprietary implementations ◦ Oracle Spatial ◦ MS SQL Server 2008 and later ◦ ArcSDE (now ArcGIS Server Basic)  Wraps spatial or non-spatial DBs for use in ESRI software, getting good performance  Some support in OS GIS, but…  In general, not useful if you need to use non-ESRI software with the DB

SDI stack: GeoServers  A variety of OS and proprietary technologies support OGC standards for geoservices

SDI stack: GeoServers  MapServer (OS). Excellent performance, quality and symbology for Map services (image)  GeoServer (OS). Excellent performance and support for Feature services (data), great integration with PostGIS, even for remote editing. Easy administration. ◦ Used in GeoNode

SDI stack: Catalog Servers  Geonetwork (OS) ◦ Reference OGC implementation for catalog service ◦ Includes powerful Web metadata editing and administration ◦ Extensively used in catalog networks (UN, many SDIs) ◦ Used in GeoNode

 ESRI’s Geoportal Server ◦ Free and Open source, but… ◦ Meant to be used with ArcGIS Online and ArcGIS for Server SDI stack: Catalog Servers

 Intergraph’s SDI Portal – Geospatial Portal ◦ Bundles OpenLayers with GeoExt UI to provide ready-to-use components SDI stack: Web clients

 Usually combine: ◦ A live-map component (map viewer) which can:  Display data from standard Web geoservices  Navigate, query data…  Sometimes provide data editing ◦ Web user-interface UI technology for legend, search, symbology, tools, etc.  Today these are based on AJAX, Flex, Silverlight or other highly interactive framework for Web apps SDI stack: Web clients

 OpenLayers (OS) ◦ Most used map viewer ◦ Supports Google Maps and other map providers ◦ Excellent performance, using tiled images ◦ Supports feature data, editing with WFS-T ◦ Used in GeoNode via the GeoExplorer component SDI stack: Web clients

 GeoExt (OS) ◦ Based on JavaScript toolkit ExtJS, applied to Web geoportals ◦ Highly interactive table of contents, menus, tools, dialogs, etc. ◦ Integrates OpenLayers. It easily extended and customized ◦ Used in GeoNode via the GeoExplorer SDI stack: Web clients

 ESRI Web Mapping API ◦ Bundles ESRI’s map components with JavaScript, Flex or Silverlight UI. Includes Web editing functionality. SDI stack: Web clients

 GIS (Geographical Information Systems) ◦ Heavy-duty data analysis and processing are still run in desktop applications ◦ All of them support OGC standards, so they can use Web geoservices and catalogs as source for data (but compliance and compatibility is not always good) ◦ The best practice is not to download data locally, but work with remote data that is official and up-to-date ◦ The SDI model, together with GIS, can also be used to implement information systems within organizations, sharing common data and metadata SDI stack: desktop clients

qGIS (FOSS) – We’ll use this one in the workshop

SDI stack: desktop clients uDIG (FOSS)

SDI stack: desktop clients ArcGIS

SDI stack: desktop clients GeoMedia

 Web Map Service (WMS) ◦ Get maps as images ◦ Get attribute information about features in a specific map location  Web Feature Service (WFS) ◦ Get vector data with geometry and attributes based on filters. Not meant for large data downloads! ◦ Remote feature editing (WFS-T)  Web Coverage Service (WCS) ◦ Similar to WFS, for raster data (multiband images, elevation grids - DEMs)  Catalog Service for Web (CSW) ◦ Search and read metadata ◦ Harvest metadata to other catalogs ◦ Remotely edit medatada

 Web Map Service (WMS) requests: ◦ GetCapabilities : returns the description of the service (layers, styles, coordinate systems…) as XML text apabilities bin/wms?map=snit.map&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetCapabilities ◦ GetMap: returns an image (jpeg, png) of a map composed of one or more layers, in the requested coordinate system. The layers and style (legend) are defined in the server but can be selected and changed (the legend) in the request S=MOD14A1_E_FIRE&WIDTH=640&HEIGHT=320&FORMAT=image/png&SRS=EPSG:4326 &BBOX=-180.0,-90.0,180.0, bin/mapserv?map=c:\ms4w\wms\biodiversidad_basp10v4geog50k.map&LAYERS=ARE AS_SILVESTRES_PROTEGIDAS%2CETIQUETAS&TRANSPARENT=true&VERSION=1.1.1&FOR MAT=image%2Fpng&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS=EPSG%3A97057 &BBOX= , , , & WIDTH=256&HEIGHT=256

 Web Map Service (WMS) requests: ◦ GetFeatureInfo request: returns the attributes of one or more objects of a specified layer at a certain location in the map. Meant as a ‘click-on’ query on a map viewer LAYERS=topp:tasmania_water_bodies&STYLES=&SERVICE=WMS&VERSION=1.1.1&REQUES T=GetFeatureInfo&BBOX= , , , &FEATURE_COUNT=10&HEIGHT=256&WIDTH=512&FORMAT=image/gif&INFO_FO RMAT=text/html&SRS=EPSG:4326&X=258&Y=188 The output format can be changed (plain text, XML…) to process it in our client application.

 WMS GetFeatureInfo Exercise ◦ Try it live at control.html control.html ◦ Use your Web Browser to spy on network requests and responses  Chrome: Tools /Developer Tools / Network tab  Firefox: Tools / Web Developer / Web Console

 Web Feature Service (WFS) requests: ◦ GetCapabilities : returns the description of the service (layers, filter types, coordinate systems…) as XML text GetCapabilities ◦ DescribeFeatureType: returns a description of the feature for a specified layer =DescribeFeatureType&typeName=usa:states

 Web Feature Service (WFS) requests: ◦ GetFeature : returns the features that comply with a certain condition (filter), based on location or attribute values (it is not meant for massive data dowload!) eature&typeName=usa:states&maxFeatures=10&filterhttp://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetF eature&typeName=usa:states&maxFeatures=10&filter= the_geom

GetFeature response in GML format South Carolina Test this live at

 Web Feature Service (WFS) requests for remote editing (WFS-T): ◦ LockFeature : prevents other clients from editing selected features while we work at them ◦ Transaction: creates, modifies or deletes features. Each transaction consists Insert, Update, and Delete elements, performed in order  Not all servers or clients implement these requests!

 Web Coverage Service (WCS): ◦ Similar to WMS but returns raster data with multiple band information, elevation values, etc.  Catalog Service for the Web (CSW): ◦ Provides remote access to a DB of metadata records (catalog)  DescribeRecord - returns info about the format of metadata records  GetRecords - search for records, returning record IDs  GetRecordsById - returns records, specified by their ID  Harvest (optional) - create/update metadata by asking the server to 'pull' metadata from somewhere (other CSW server, a Web folder…)  Transaction (optional) - create/edit metadata by 'pushing' the metadata to the server

 Catalog Service for the Web (CSW): ◦ GetCapabilities and DescribeRecord examples: Capabilities cribeRecord ◦ GetRecords example : VICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA= 2005/gmd&CONSTRAINTLANGUAGE=CQL_TEXT&RESULTTYPE=results&T YPENAMES=csw:Record A constraint or filter is usually added to get only the metadata that we are searching (for instance, containing certain words, or for a certain geographic area)

The CSW service, through the Harvest request, allows the federation and integration of metadata catalogs, automatically collecting metadata from some catalogs to another (for instance, to specific organizations to a national catalog) Basic SDI Services Metadat a DB National Catalog Server Web or desktop client Local/entity Catalog Server Harvest Metadata editor

MetadataFile Metadata Web Service Metadata Geoportal, GIS client Main Catalog Entity metadata Search CSW request CSW Result Metadata Entity metadata Metadata Entity data Archivo File Viewer Add WMS layer Geodata Servidor entidad WMS request WMS result (image) Add WFS layer Query WFS request WFS result (data) Data visualization Data download Basic SDI Services How they all work together… (Animation)

 Direct file download ◦ Use it for geodata download instead of WFS or WCS, which should be only for limited query ◦ Use it for documents and other files that cannot be accessed through standard Web services ◦ The download address can be written in the metadata, as we do for Web services  News and alerts (RSS, GeoRSS, Atom…) ◦ Conventional Web clients can get updated messages and news ◦ Users can subscribe to receive them in a variety of readers

 Metadata is a formal data documentation and it is critical to data discovery  A metadata record is a set of information fields, usually presented as an XML document, which captures the basic characteristics of data as an information resource  It represents the who, what, when, where, why and how of the resource

Metadata as seen in Geonetwork What is "Metadata"?

Metadata as seen in ESRI’s ArcCatalog

 Organize and maintain an organization's internal investment in spatial data  Provide information about an organization's data holdings to data catalogs, clearinghouses, and portals (it is the entry point of an SDI and for each SDI node)  Provide information to process and interpret data received through a transfer from an external source

 Facilitates data maintenance  Facilitate data discovery  Enables reuse of data  Informs potential users of inappropriate uses of a dataset

 Exchanging records -> Use standards!  Creating metadata takes time and patience -> Use templates and tools that automatically extract or set metadata (like GeoNode). Imperfect metadata is better than none  Maintaining metadata is essential, especially for Web services and downloads (beware of URL changes!) -> set up processes or tools for automatic checkups

 Some standards define the metadata content (fields, values, mandatory v. optional) ◦ The most comprehensive is ISO  From this standards, custom profiles can be defined for specific countries, themes, etc.  ‘Old’ metadata standards like FGDC or Dublin Core are now profiles of ISO  Other standards define appropriate exchange formats ◦ The most accepted is ISO 19139, which defines an XML-based format

Important for data search

Information in abstract: data model – Entity types and their attributes and the domains from which attribute values may be assigned – The names and definitions of features, attributes, and attribute values

How to access the information? Multiple transfer options are possible Contact for physical media Fees

Positional and attribute accuracy Completeness and consistency, Sources of information Methods used to produce the data Appropriate uses (scales)

Language, date, update status Who is responsible for the metadata?

 Use constraint statements can be crafted to express scale, geographic, or temporal limitations to the data  Liability statements should be written by legal staff to ensure that the legal requirements for use of the data are fully outlined  In general, it is far better to publish your dataset with limitations within your metadata than to later attempt to generate them in response to an inquiry or lawsuit. Metadata content: Access and use constraints

 Use constraints and disclaimers ◦ Web clients typically include liability limitations (disclaimers) and licenses that users must accept ◦ But geoservices can be accessed directly!  Therefore, use constraints must be specified in the metadata and GetCapabilities response  And services must be protected from non- authorized access ◦ Within an organization: generic IP filtering and firewalls ◦ External services: need an authentication mechanism

 There are generic mechanisms for Service- Oriented Architectures ◦ Based on XML standards like SAML and XACML – However, these standards are not easily used in SDIs because Current geoservices do not use SOAP for passing information Server and client software should be modified to comply

 Specific SDI solutions have been proposed and used  Existing clients and server don’t have to be modified, only ‘intercepting’ components need to be added  Access can be controlled for specific layers, scales, features and attributes See ra_V2.pdf ra_V2.pdf

 In practice, however, most systems use simple HTTP authentication that provides a simple on/off swith for services depending on the user credentials  Web browsers have a built-in implementation to ask for user and password

 Gazetteer services (search location by name), based on WFS-G profile

 Web Processing Service (WPS). Standard for discovering and calling remote processing modules ◦ Used for typical GIS analysis like spatial analysis, running distributed models ◦ Used for operations that are not easy to standardize in a specific Web service  Spatial Coordinates transformation  Geocoding (address finding) See, for instance: rvice=WPS&request=GetCapabilities rvice=WPS&request=GetCapabilities

Current trends in SDI technology Tiled maps Cached and tiled Map Services (WMTS, WMS- C…)

 Static maps can be pre-calculated and tiled at different scales, like a raster pyramid ◦ Tiles can be directly server through tile services like WMTS, WMS-C, Google Earth… ◦ Pre-calculated tiles can be used to quickly respond to conventional WMS requests ◦ An example is the GeoWebCache technology, used by GeoNode, which can be integrated with regular WMS servers like GeoServer Current trends in SDI technology Tiled maps

Current trends in SDI technology Tiled maps

Current trends in SDI technology Tiled maps  A problem with tiled map providers like Google, Bind, OpenStreetMap, is that they only provide tiles with a specific ad-hoc coordinate system (used to be EPSG:900913, now it is officially EPSG:3857)  It is difficult to overlay this data layer on other coordinate systems

Remote on-line editing with WFS-T Current trends in SDI technology

Remote collaborative editing (OpenStreetMap) Current trends in SDI technology

Data capture and visualization in mobile devices

Remote geoprocesses (coordinate transformation, geocoding) with WPS Current trends in SDI technology

Real-time data integration (RSS, Common Alert Protocol) Current trends in SDI technology

Standard Web services for reading and controling remote sensors (SWE, SOS…) Current trends in SDI technology

Integration with SOLAP data mining Current trends in SDI technology

SDI Web services in the cloud Current trends in SDI technology

New data sources (LiDAR, georeferenced Video & street-level images) Current trends in SDI technology