1. Salvador Bayarri Consultant, World Bank sbayarri@gmail.com

2.  The SDI architecture model  Software components: the SDI stack  Basic services  Metadata  SDI access control  Current trends in SDI technology

3.  SDI uses a three-tier service-oriented model Source: mapit.biz Standardized

4. Same thing with a component-oriented view: Clients Middleware Servers FeaturesCoverages Metadata update Catalogs Geo-processing and catalog Services Content Repositories Other data e.g., administrative, statistical, env. reporting Service chaining: search, display, access, e-commerce, …. Direct data access Access to transformed data, pictures, maps, reports, multi-media content Metadata search and retrieval for data and servicesGIS

5.  The use of a communication layer based on standardized Web services ensures: ◦ Transparent access to data, regardless of format, technology… but we still need to harmonize the data content ◦ Interoperability between different brands of servers and clients ◦ Distributability of components in a network. The SDI philosophy is that each responsible organization maintains and publishes its datasets. Multiple clients, catalogs, etc. are possible

7.  Spatial DB = Relational DB + … ◦ Geometry representation. Standard is SFS (Simple Feature Specification) ◦ Spatial Index. Speeds up search and operators, sorting objects by location ◦ Spatial operators. Can be invoked in SQL sentences to measure, perform geometric operations (e.g. intersection, reprojection), create new objects, etc. Essential for data quality assurance.  Needed for large datasets, many simultaneous accesses (e.g. concurrent editing)  Relational databases also used for metadata  Files still used for raster data (images, elevation grids)

8.  Example of spatial table Encoded geometry field

9.  Simple Feature Specification (OGC) for SQL SDI stack: Spatial Databases

10.  Well-Known Binary representation for simple features (OGC)

11.  Open-source (OS) implementations ◦ SpatiaLite (extension to sqLite), supported by qGIS ◦ MySQL spatial extensions (not fully standard) ◦ PostGIS (extension to PostgreSQL)  Excellent support for standards  Widely supported in OS GIS, ArcGIS, Intergraph…  Used by GeoNode  Friendly administration tools (pgAdmin)  Interesting extensions (pgRoute for routing) The whole UK Ordnance Survey is stored in PostGIS (£1,000,000 savings compared to Oracle Spatial), as well as the national topographic data of France for the Institut Géographique National

12.  Proprietary implementations ◦ Oracle Spatial ◦ MS SQL Server 2008 and later ◦ ArcSDE (now ArcGIS Server Basic)  Wraps spatial or non-spatial DBs for use in ESRI software, getting good performance  Some support in OS GIS, but…  In general, not useful if you need to use non-ESRI software with the DB

13. SDI stack: GeoServers  A variety of OS and proprietary technologies support OGC standards for geoservices

14. SDI stack: GeoServers  MapServer (OS). Excellent performance, quality and symbology for Map services (image)  GeoServer (OS). Excellent performance and support for Feature services (data), great integration with PostGIS, even for remote editing. Easy administration. ◦ Used in GeoNode

15. SDI stack: Catalog Servers  Geonetwork (OS) ◦ Reference OGC implementation for catalog service ◦ Includes powerful Web metadata editing and administration ◦ Extensively used in catalog networks (UN, many SDIs) ◦ Used in GeoNode

16.  ESRI’s Geoportal Server ◦ Free and Open source, but… ◦ Meant to be used with ArcGIS Online and ArcGIS for Server SDI stack: Catalog Servers

17.  Intergraph’s SDI Portal – Geospatial Portal ◦ Bundles OpenLayers with GeoExt UI to provide ready-to-use components SDI stack: Web clients

18.  Usually combine: ◦ A live-map component (map viewer) which can:  Display data from standard Web geoservices  Navigate, query data…  Sometimes provide data editing ◦ Web user-interface UI technology for legend, search, symbology, tools, etc.  Today these are based on AJAX, Flex, Silverlight or other highly interactive framework for Web apps SDI stack: Web clients

19.  OpenLayers (OS) ◦ Most used map viewer ◦ Supports Google Maps and other map providers ◦ Excellent performance, using tiled images ◦ Supports feature data, editing with WFS-T ◦ Used in GeoNode via the GeoExplorer component SDI stack: Web clients

20.  GeoExt (OS) ◦ Based on JavaScript toolkit ExtJS, applied to Web geoportals ◦ Highly interactive table of contents, menus, tools, dialogs, etc. ◦ Integrates OpenLayers. It easily extended and customized ◦ Used in GeoNode via the GeoExplorer SDI stack: Web clients

21.  ESRI Web Mapping API ◦ Bundles ESRI’s map components with JavaScript, Flex or Silverlight UI. Includes Web editing functionality. SDI stack: Web clients

22.  GIS (Geographical Information Systems) ◦ Heavy-duty data analysis and processing are still run in desktop applications ◦ All of them support OGC standards, so they can use Web geoservices and catalogs as source for data (but compliance and compatibility is not always good) ◦ The best practice is not to download data locally, but work with remote data that is official and up-to-date ◦ The SDI model, together with GIS, can also be used to implement information systems within organizations, sharing common data and metadata SDI stack: desktop clients

23. qGIS (FOSS) – We’ll use this one in the workshop

24. SDI stack: desktop clients uDIG (FOSS)

25. SDI stack: desktop clients ArcGIS

26. SDI stack: desktop clients GeoMedia

27.  Web Map Service (WMS) ◦ Get maps as images ◦ Get attribute information about features in a specific map location  Web Feature Service (WFS) ◦ Get vector data with geometry and attributes based on filters. Not meant for large data downloads! ◦ Remote feature editing (WFS-T)  Web Coverage Service (WCS) ◦ Similar to WFS, for raster data (multiband images, elevation grids - DEMs)  Catalog Service for Web (CSW) ◦ Search and read metadata ◦ Harvest metadata to other catalogs ◦ Remotely edit medatada

28.  Web Map Service (WMS) requests: ◦ GetCapabilities : returns the description of the service (layers, styles, coordinate systems…) as XML text http://neowms.sci.gsfc.nasa.gov/wms/wms?version=1.1.1&service=WMS&request=GetC apabilities http://www.snitcr.org/cgi- bin/wms?map=snit.map&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetCapabilities ◦ GetMap: returns an image (jpeg, png) of a map composed of one or more layers, in the requested coordinate system. The layers and style (legend) are defined in the server but can be selected and changed (the legend) in the request http://neowms.sci.gsfc.nasa.gov/wms/wms?VERSION=1.1.1&REQUEST=GetMap&LAYER S=MOD14A1_E_FIRE&WIDTH=640&HEIGHT=320&FORMAT=image/png&SRS=EPSG:4326 &BBOX=-180.0,-90.0,180.0,90.0 http://196.40.0.82/cgi- bin/mapserv?map=c:\ms4w\wms\biodiversidad_basp10v4geog50k.map&LAYERS=ARE AS_SILVESTRES_PROTEGIDAS%2CETIQUETAS&TRANSPARENT=true&VERSION=1.1.1&FOR MAT=image%2Fpng&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS=EPSG%3A97057 &BBOX=247206.87468627,1128978.9530588,427828.99937255,1309601.0777451& WIDTH=256&HEIGHT=256

29.  Web Map Service (WMS) requests: ◦ GetFeatureInfo request: returns the attributes of one or more objects of a specified layer at a certain location in the map. Meant as a ‘click-on’ query on a map viewer http://demo.opengeo.org/geoserver/wms?LAYERS=topp:tasmania_water_bodies&QUERY_ LAYERS=topp:tasmania_water_bodies&STYLES=&SERVICE=WMS&VERSION=1.1.1&REQUES T=GetFeatureInfo&BBOX=141.5115,-43.933,150.8015,- 39.288&FEATURE_COUNT=10&HEIGHT=256&WIDTH=512&FORMAT=image/gif&INFO_FO RMAT=text/html&SRS=EPSG:4326&X=258&Y=188 The output format can be changed (plain text, XML…) to process it in our client application.

30.  WMS GetFeatureInfo Exercise ◦ Try it live at http://openlayers.org/dev/examples/getfeatureinfo- control.html http://openlayers.org/dev/examples/getfeatureinfo- control.html ◦ Use your Web Browser to spy on network requests and responses  Chrome: Tools /Developer Tools / Network tab  Firefox: Tools / Web Developer / Web Console

31.  Web Feature Service (WFS) requests: ◦ GetCapabilities : returns the description of the service (layers, filter types, coordinate systems…) as XML text http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request= GetCapabilities ◦ DescribeFeatureType: returns a description of the feature for a specified layer http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request =DescribeFeatureType&typeName=usa:states

32.  Web Feature Service (WFS) requests: ◦ GetFeature : returns the features that comply with a certain condition (filter), based on location or attribute values (it is not meant for massive data dowload!) http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetF eature&typeName=usa:states&maxFeatures=10&filterhttp://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetF eature&typeName=usa:states&maxFeatures=10&filter= the_geom -89.92773475911 32.386718678814 -89.04882850911 33.265624928814

33. GetFeature response in GML format -81.759758 33.195232000000004... South Carolina 3486703.0 Test this live at http://openlayers.org/dev/examples/getfeature-wfs.html http://openlayers.org/dev/examples/getfeature-wfs.html

34.  Web Feature Service (WFS) requests for remote editing (WFS-T): ◦ LockFeature : prevents other clients from editing selected features while we work at them ◦ Transaction: creates, modifies or deletes features. Each transaction consists Insert, Update, and Delete elements, performed in order  Not all servers or clients implement these requests!

35.  Web Coverage Service (WCS): ◦ Similar to WMS but returns raster data with multiple band information, elevation values, etc.  Catalog Service for the Web (CSW): ◦ Provides remote access to a DB of metadata records (catalog)  DescribeRecord - returns info about the format of metadata records  GetRecords - search for records, returning record IDs  GetRecordsById - returns records, specified by their ID  Harvest (optional) - create/update metadata by asking the server to 'pull' metadata from somewhere (other CSW server, a Web folder…)  Transaction (optional) - create/edit metadata by 'pushing' the metadata to the server

36.  Catalog Service for the Web (CSW): ◦ GetCapabilities and DescribeRecord examples: http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=Get Capabilities http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=Des cribeRecord ◦ GetRecords example : http://www.fao.org/geonetwork/srv/en/csw?REQUEST=GetRecords&SER VICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/ 2005/gmd&CONSTRAINTLANGUAGE=CQL_TEXT&RESULTTYPE=results&T YPENAMES=csw:Record A constraint or filter is usually added to get only the metadata that we are searching (for instance, containing certain words, or for a certain geographic area)

37. The CSW service, through the Harvest request, allows the federation and integration of metadata catalogs, automatically collecting metadata from some catalogs to another (for instance, to specific organizations to a national catalog) Basic SDI Services Metadat a DB National Catalog Server Web or desktop client Local/entity Catalog Server Harvest Metadata editor

38. MetadataFile Metadata Web Service Metadata Geoportal, GIS client Main Catalog Entity metadata Search CSW request http://catalog.smi-q.gob.ec CSW Result Metadata Entity metadata Metadata Entity data Archivo File http://server.dep.gob.ec/files/data.dat Viewer Add WMS layer Geodata Servidor entidad WMS request http://server.dep.gob.ec/maps?map1 WMS result (image) Add WFS layer Query WFS request http://server.dep.gob.ec/wfs?layer1 WFS result (data) Data visualization Data download Basic SDI Services How they all work together… (Animation)

39.  Direct file download ◦ Use it for geodata download instead of WFS or WCS, which should be only for limited query ◦ Use it for documents and other files that cannot be accessed through standard Web services ◦ The download address can be written in the metadata, as we do for Web services  News and alerts (RSS, GeoRSS, Atom…) ◦ Conventional Web clients can get updated messages and news ◦ Users can subscribe to receive them in a variety of readers

40.  Metadata is a formal data documentation and it is critical to data discovery  A metadata record is a set of information fields, usually presented as an XML document, which captures the basic characteristics of data as an information resource  It represents the who, what, when, where, why and how of the resource

41. Metadata as seen in Geonetwork What is "Metadata"?

42. Metadata as seen in ESRI’s ArcCatalog

43.  Organize and maintain an organization's internal investment in spatial data  Provide information about an organization's data holdings to data catalogs, clearinghouses, and portals (it is the entry point of an SDI and for each SDI node)  Provide information to process and interpret data received through a transfer from an external source

44.  Facilitates data maintenance  Facilitate data discovery  Enables reuse of data  Informs potential users of inappropriate uses of a dataset

45.  Exchanging records -> Use standards!  Creating metadata takes time and patience -> Use templates and tools that automatically extract or set metadata (like GeoNode). Imperfect metadata is better than none  Maintaining metadata is essential, especially for Web services and downloads (beware of URL changes!) -> set up processes or tools for automatic checkups

46.  Some standards define the metadata content (fields, values, mandatory v. optional) ◦ The most comprehensive is ISO 19115  From this standards, custom profiles can be defined for specific countries, themes, etc.  ‘Old’ metadata standards like FGDC or Dublin Core are now profiles of ISO 19115  Other standards define appropriate exchange formats ◦ The most accepted is ISO 19139, which defines an XML-based format

47. Important for data search

48. Information in abstract: data model – Entity types and their attributes and the domains from which attribute values may be assigned – The names and definitions of features, attributes, and attribute values

50. How to access the information? Multiple transfer options are possible Contact for physical media Fees

51. Positional and attribute accuracy Completeness and consistency, Sources of information Methods used to produce the data Appropriate uses (scales)

53. Language, date, update status Who is responsible for the metadata?

54.  Use constraint statements can be crafted to express scale, geographic, or temporal limitations to the data  Liability statements should be written by legal staff to ensure that the legal requirements for use of the data are fully outlined  In general, it is far better to publish your dataset with limitations within your metadata than to later attempt to generate them in response to an inquiry or lawsuit. Metadata content: Access and use constraints

55.  Use constraints and disclaimers ◦ Web clients typically include liability limitations (disclaimers) and licenses that users must accept ◦ But geoservices can be accessed directly!  Therefore, use constraints must be specified in the metadata and GetCapabilities response  And services must be protected from non- authorized access ◦ Within an organization: generic IP filtering and firewalls ◦ External services: need an authentication mechanism

56.  There are generic mechanisms for Service- Oriented Architectures ◦ Based on XML standards like SAML and XACML – However, these standards are not easily used in SDIs because Current geoservices do not use SOAP for passing information Server and client software should be modified to comply

57.  Specific SDI solutions have been proposed and used  Existing clients and server don’t have to be modified, only ‘intercepting’ components need to be added  Access can be controlled for specific layers, scales, features and attributes See http://www.eurogeographics.org/sites/default/files/070504_SecurityInSDI_conter ra_V2.pdf http://www.eurogeographics.org/sites/default/files/070504_SecurityInSDI_conter ra_V2.pdf http://52north.org/communities/security/

58.  In practice, however, most systems use simple HTTP authentication that provides a simple on/off swith for services depending on the user credentials  Web browsers have a built-in implementation to ask for user and password

59.  Gazetteer services (search location by name), based on WFS-G profile

60.  Web Processing Service (WPS). Standard for discovering and calling remote processing modules ◦ Used for typical GIS analysis like spatial analysis, running distributed models ◦ Used for operations that are not easy to standardize in a specific Web service  Spatial Coordinates transformation  Geocoding (address finding) See, for instance: http://www.cartociudad.es/wps/WebProcessingService?se rvice=WPS&request=GetCapabilities http://www.cartociudad.es/wps/WebProcessingService?se rvice=WPS&request=GetCapabilities

61. Current trends in SDI technology Tiled maps Cached and tiled Map Services (WMTS, WMS- C…)

62.  Static maps can be pre-calculated and tiled at different scales, like a raster pyramid ◦ Tiles can be directly server through tile services like WMTS, WMS-C, Google Earth… ◦ Pre-calculated tiles can be used to quickly respond to conventional WMS requests ◦ An example is the GeoWebCache technology, used by GeoNode, which can be integrated with regular WMS servers like GeoServer Current trends in SDI technology Tiled maps

63. Current trends in SDI technology Tiled maps

64. Current trends in SDI technology Tiled maps  A problem with tiled map providers like Google, Bind, OpenStreetMap, is that they only provide tiles with a specific ad-hoc coordinate system (used to be EPSG:900913, now it is officially EPSG:3857)  It is difficult to overlay this data layer on other coordinate systems

65. Remote on-line editing with WFS-T Current trends in SDI technology

66. Remote collaborative editing (OpenStreetMap) Current trends in SDI technology

67. Data capture and visualization in mobile devices

68. Remote geoprocesses (coordinate transformation, geocoding) with WPS Current trends in SDI technology

69. Real-time data integration (RSS, Common Alert Protocol) Current trends in SDI technology

70. Standard Web services for reading and controling remote sensors (SWE, SOS…) Current trends in SDI technology

71. Integration with SOLAP data mining Current trends in SDI technology

72. SDI Web services in the cloud Current trends in SDI technology

73. New data sources (LiDAR, georeferenced Video & street-level images) Current trends in SDI technology