1. Marten Hogeweg Satish Sankaran
Building Effective Data Portals Using Standards Compliant ArcGIS Platform
Marten Hogeweg
Satish Sankaran

2. Finding Data

3. Spatial Data Infrastructure Portals
Data Portals
Spatial Data Infrastructure
Portals
Registry
catalogs
Metadata catalogs
Data catalogs
Data Clearinghouse
Emphasis on standards versus emphasis on “open”.
In its broadest sense, a geospatial data clearinghouse, is a distributed system of agency servers located on the Internet that contain field-level descriptions of available and planned digital spatial data, applications, and services. As such, the clearinghouse may not be a central repository of information, but rather a routing station which forwards user requests to the source of the desired information
A registry is a list of items with pointers for where to find the items, like the index on a database table or the card catalog for a library. A repository stores the actual items, like a database table itself or a library's shelves of books. If you lose a registry, the items still exist; you just may need to reindex them. If you lose a repository, the items are gone.“
A metadata registry is a central location in an organization where metadata definitions are stored and maintained in a controlled method. ebrim=registry info model
A metadata repository is the database where metadata is stored. The registry also adds relationships with related metadata types.
Repository
Open Data
Geo-Portals
Geohub

4. Open Data

5. Geoportals & Metadata Catalogs
Metadata ctalogs … intent was not to provide the full-on portal experience.
Early view of the geoportal. People could register their metadata and data/services.
Or register their geospatial clearinghouses and the top node/portal could harvest metadata or in some cases federate searches to these nodes.

6. Spatial Data Infrastructure (SDI)
Coined in 1993 by the US National Research Council
Global Spatial Data Infrastructure, GSDI Cookbook
An SDI denotes the relevant base collection of technologies, policies and institutional arrangements that facilitate the availability of and access to spatial data.
It provides a basis for spatial data discovery, evaluation, and application for users and providers within all levels of government, the commercial sector, the non-profit sector, academia and by citizens in general.
A SDI should enable the discovery and delivery of spatial data from a data repository, via a spatial service provider, to a user. As mentioned earlier it is often wished that the data provider is able to update spatial data stored in a repository. Hence, the basic software components of an SDI are:[3]
Software client - to display, query, and analyse spatial data (this could be a browser or a desktop GIS)
Catalogue service - for the discovery, browsing, and querying of metadata or spatial services, spatial datasets and other resources
Spatial data service - allowing the delivery of the data via the Internet
Processing services - such as datum and projection transformations
(Spatial) data repository - to store data, e.g., a spatial database
GIS software (client or desktop) - to create and update spatial data

7. Progression of Geospatial Initiatives led by FGDC
In 1990, the OMB established the FGDC, an interagency committee composed of representatives from the Executive Office of the President and Cabinet‐level and independent Federal agencies.  Over the last two decades, the FGDC led several geospatial initiatives that helped form the vision, and provided the foundational policies, strategies, and content for, the Geospatial Platform.   These FGDC‐led initiatives include: 1) The Geospatial Clearinghouse Network4 ; 2) Geospatial One Stop (GOS); and 3) The Geospatial Line of Business (Geospatial LoB).  The development of the Business Plan and evolution of the operational Geospatial Platform provides an opportunity to document lessons learned from preceding geospatial initiatives and incorporate them into the development of the Geospatial Platform.  Descriptions of these preceding initiatives and lessons learned are included in Appendix F.  Figure 1 summarizes the progression of the initiatives.
The Office of Management and Budget (OMB) established the FGDC in 1990 and rechartered the committee in its August 2002 revision of Circular A-16, "Coordination of Geographic Information and Related Spatial Data Activities." The FGDC is a 32 member interagency committee composed of representatives from the Executive Office of the President, and Cabinet level and independent Federal agencies. The Secretary of the Department of the Interior chairs the FGDC, with the Deputy Director for Management, Office of Management and Budget (OMB) as Vice-Chair. Numerous stakeholder organizations participate in FGDC activities representing the interests of state and local government, industry, and professional organizations. Learn more about the organization of the FGDC.

8. SDI https://en.wikipedia.org/wiki/Spatial_data_infrastructure
In line with its data and information policy, the Copernicus programme provides users with free, full and open access to environmental data.
Depending on their needs, users can obtain these data either from the Copernicus services or directly from the Copernicus satellites. No registration is required for discovery and view services while it is a prerequisite to download Sentinel data. Registration is free of charge.
A central part of GEO’s Mission is to build the Global Earth Observation System of Systems (GEOSS). GEOSS is a set of coordinated, independent Earth observation, information and processing systems that interact and provide access to diverse information for a broad range of users in both public and private sectors.

9. SDI / MSDI around the world

10. Open Data - (typically) All about Supporting Open Government
Agencies shall take the following actions to improve the management of information resources throughout the information’s life cycle and reinforce the government’s presumption in favor of openness. (taken from the Policy Requirements section)
Data
Standards
Machine
Readable
Open
Licenses
Interoperability
Engage
Customers
Enterprise
Data Inventory
Open
Formats
M — Memorandum for the Heads of Executive Departments and Agencies
 Open Data Policy—Managing Information as an Asset
III. Policy Requirements1. Collect or create information in a way that supports downstream information processing and dissemination activities
a. Use machine-readable and open formats
b. Use data standards
c. Ensure information stewardship through the use of open licenses
d. Use common core and extensible metadata
2. Build information systems to support interoperability and information accessibility
3. Strengthen data management and release practices
a. Create and maintain an enterprise data inventory
b. Create and maintain a public data listing
c. Create a process to engage with customers to help facilitate and prioritize data release
d. Clarify roles and responsibilities for promoting efficient and effective data release practices
4. Strengthen measures to ensure that privacy and confidentiality are fully protected and that data are properly secured
5. Incorporate new interoperability and openness requirements into core agency processes
Information
Processing
Privacy
Accessibility
Confidentiality 
Public
Data Listing
Metadata

11. Open Data Portals
Has your organization realized a return on investment for your open data efforts? When it was first introduced, open data was defined as the innovation currency from which to best understand how to improve accountability and transparency. We were promised greater civic engagement and a boost to local economies. Over the last few years we learned a lot about the benefits of opening data; not just for the public but for governments as well. Opening up your data inward is helping to break down silos within organizations. Open data has become a hub for productivity and efficiency.

12. 111,44,7 and then michigan
dataportals.org/
p://dataportals.org/

13. What makes a Data Portal?
Datasets
Metadata
Services
Indexing and Search capabilities
&
Other stuff
Harvesting , federated searches
Look at online’s functionality

14. Standards & Specifications in the World of Data Portals
Data Format Standards
Shapefiles , CSV ,
GeoJSON, GML
Atom / GEORSS
Image Formats
Service Standards
WMS, WMTS, WFS
WCS, KML, CS-W
WPS, KML, SOS
Metadata & Catalog Standards
Dublin Core
ISO , FGDC ,
RDF , Linked Data
CS-W , OpenSearch
DCAT / GeoDCAT -AP
A list of portal solutions that we are currently evaluating:
ArcGIS Online
ArcGIS Open Data
Portal for ArcGIS Server (on-premises)
Socrata and Mondara
Google Maps Engine and Gallery
Other portals we haven't yet considered:
DataHub
CKAN
DKAN
OpenGeoportal
Not really portals but work well for serving flat files:
Dropbox
Box
FTP (on-premises)

15. ArcGIS - Building Geospatial Portals
ArcGIS Online
ArcGIS Online + Open Data
ArcGIS for Server with Portal for ArcGIS
ArcGIS for Inspire
Esri GeoPortal Server
Differentiate online/portal for arcgis with general sdi portals

16. Building Effective Data Portals
@martenhogeweg

17. Communities Become Smarter When They Share Data
Leveraging the Value of Shared Information
Local
Government
States
Citizens
NGOs
Federal
Agencies
Schools
Businesses
Open Services
Open Data

18. Data Sharing Is Key to Success
Share as much as possible
Provide access to authoritative data
Provide access to current data
Make data widely accessible
Ensure data is machine readable
Use open formats and API
Share
Discover
Evaluate
Apply
Improve
Based on left out the licensing part (not our decision to decide on licensing government data, plus some companies live of value added data services: good for them)

19. If You Can’t Find It, You Can’t Use It
arcgis.com
geoplatform.gov
data.gov
data.gov.uk …
Key to transparency and efficiency – GIS community has had a jumpstart on data sharing over the years and is a model for the current broader government efforts. However, GIS community needs to now step it up to evolve to a platform and remain a leader
Data Portals Improve Access . . .

20. Open Data Portal Context

21. Geospatial Platform Geoplatform.gov Data.gov … NASA GSA Census FCC DOI
The government of the USA has a long track record of building national spatial data infrastructures that started in the mid-nineties. The current generation of this NSDI is referred to as the Geospatial Platform with a portal at
In addition to the NSDI, there is now a growing trend of generic ‘open data sharing’ where many non-spatial datasets are being made available in usable formats.
DOT
NOAA
DOJ
EPA
DHHS
DHS

22. GeoSUR (Latin America)
Venezuela
INPE
Paraguay
Brazil
. . .
GeoSUR
Argentina
An example of a regional SDI is GeoSUR. GeoSUR connects national SDI in latin America and the caribean.
Panama
Uruguay
Mexico
Chile
Ecuador
Peru
Bolivia

23. Group on Earth Observations
SAEOS
INPE
PANGAEA
GCMD
ArcGIS Online
GEO
SeaDataNet
An example of a global SDI is the Group on Earth Obervations (GEO), which positions itself as a network of networks. With this they mean the connection that is made between existing global data sharing networks that focus on a specific domain, such as the World Meteorological Organization’s (WMO) Weather Information System (WIS). Apart from connecting these networks, GEO provides a single view into all of these networks through a portal (
Such a portal is a common component of an SDI. It provides a so-called ‘one-stop shop’ for data users to discover the data and services provided by the SDI. The role of this portal is essentially to fill a gap between the many individual contributor sites (where one has to visit many websites to find the right resources) and common web search engines (where the specialized geospatial resources are lost between the far larger amount of general web content).
EEA-SDI
EuroGEOSS
WMO WIS
IOOS
GSFC-DAAC
EnviroGRIDS
GBIF

24. The Issue IS the Issue
… not the portal

25. Primary Focus Area for Data Sharing Initiatives
National Security
Environment
Economic
SDI are created to support many missions, but generally speaking, these missions fall into 3 main focus areas: Economy, National Security, and Environment.

26. National Intelligence
National Security
Prepare, Respond and Recover
Requires:
Make Better Decisions
Faster Decisions
Ensure Unity in Action
Provide:
Common Operating Pictures
Incident Response Plans
Standard Map Products
Shared Situational Awareness
The goal of implementing SDI for National Security is to help prepare, respond and recover from any national emergency. In these three areas, time is the critical factor.
National Security SDI aims to facilitate detecting threats, being able to analyse the situation, and then act on this in unity.
What we see in national security SDI are groupings of organizations like national intelligence, disaster management, first responders, law enforcement, defence, health, private enterprises, and utilities.
For example: the Virtual USA is an innovative information-sharing initiative—developed in collaboration with the emergency response community and state and local governments across the nation—that helps first responders at all levels of government communicate during emergencies. Virtual USA links disparate tools and technologies in order to share the location and status of critical assets and information—such as power and water lines, flood detectors, helicopter-capable landing sites, emergency vehicle and ambulance locations, weather and traffic conditions, evacuation routes, and school and government building floor plans—across federal, state/local governments.
(From DHS brochure) Emergency responders—emergency medical services, fire-rescue personnel, and law enforcement officers—need to share vital data or voice information across disciplines and jurisdictions to successfully respond to day-to-day incidents and large-scale emergencies. Many people assume that emergency response agencies across the Nation are already interoperable. In actuality, emergency responders often cannot talk to some parts of their own agencies—let alone communicate with agencies in neighboring cities, counties, or states.
National Intelligence
Disaster Management
Fire / Rescue
Law
Enforcement
Defense
Health
Private
Enterprise
Utilities

27. Economic Innovation, Transparency, Accountability, & Austerity
Requires:
Promoting Innovation
Responding to FOIA request
Justifying Policy
Provide:
Open Data Access
Transparency Websites
App Challenges
For government organizations economic SDI focuses on innovation, transparency, accountability, and austerity. For private industry the focus is on increasing revenue. The types of data and organizations involved are census, commerce, planning, finance, land administration, health, industry, and academia
Open Government Focus Areas include:
Transparency – provide public with information about their government’s activities
Participation – expand citizen’s opportunities to express their views about policy beyond elections
Collaboration – citizens help shape the government’s agenda
Census
Commerce
Planning
Finance
Land Administration
Health
Industry
Education

28. Environment Resiliency, Measure, Report, Verify & Predict Requires:
Make Better Decisions
Enforce Policy
Provide:
Reports on the Environment
Indicators
The environmental SDI organizations are focused on increasing resiliency, measuring, predicting, and reporting impacts. Another critical mission is verifying information and enforcing laws for compliance.
Environmental SDI typically involves data themes like Foundation data, Land Cover/Land Use, Natural resources like geology, statistics, health, and industry data.
Foundation Data
Land Cover/Use
Geology
Environmental
Statistics
Health
Industry

29. Open Data Portals Evolve
… or do they?

30. Evolving Views on Data Portals
Web GIS
Evolving Views on Data Portals
GIS Portals
Services Oriented
Focus on Content
Integration in Business Process
Apps
Clearinghouses
Application focus
IT standards
Interoperable
File Transfer
System-oriented
Data focus
Geo-standards
Top down
Ad hoc
Data focus
FTP transfer
Bottom up

31. Size Matters
… or does it?

32. 100s
Local Government
LA GeoHub – 524 (as of 5/31/2016)

33. 1,000s
State Government

34. 47,000
ArcGIS Open Datasets
46,746 – as of 5/31/2016 (from 3,091 organizations)

35. 200,000
Data.gov
182,162 as of 5/31/2016

36. 100,000,000s
NOAA, NASA, GEOSS…

37. An Effective Data Portal?

38. Experiences Common Across Catalogs
Search Results Not Useful (precision)
Search Results Not Complete (recall)
Incomplete Descriptions
Not Working Links
Catalog Not Up-to-date
BLM and BLM on Data.gov
Download NEXRAD Anyone?
Serverize Your Data!
License Your Data!
User Experience Design
GEOSS Portal
Unclear Web Site Navigation
Incomplete Descriptions
ArcGIS Online demo/demo, test/test, …
Search Results Not Useful (precision) + Search Results Not Complete (recall)
Explain picture
Up to catalog implementers
Not about searching full metadata fields
Not Working Links
GeoSUR integration with FGDC Service Status Checker
Catalog Not Up-to-date
BLM and BLM on Data.gov
Data.gov has only few hundred where one BLM site has 10x that much. Why?
Download NEXRAD Anyone?
Navigate folder structure without information
Download thousands of individual files?
Mention Landsat Image Services as alternative
Unclear Web Site Navigation
Geoplatform.gov updated website.
Some parts still in old design
Breaking navigation: Home > Search > Home

39. A Different Approach

40. I Told You So…

41. I Told You So…

42. An Effective Data Portal is Not Just a Data Portal