U.S. Department of the Interior U.S. Geological Survey Web Presence, Data Sharing, Real- time Analysis and Crowdsourcing GFSAD30 January 2015 Workshop Justin Poehnelt, Student Developer

2014 Achievements  Data products are being distributed from multiple sources.  An interactive map with multiple layers and a legend is available for public viewing.  A beginning framework for crowdsourcing training and validation data has been established.

2014 Challenges  Requirements are beginning to stabilize for croplands web application.  Custom web application requires custom architecture, design, codebase and testing.  Development has been a very iterative process.

USGS Web Portal While project updates have been posted, more significant changes will be delayed until USGS migration to new content management system. A storage solution for publicly available large files has been implemented on Flagstaff servers. Central location to point visitors.

Google Earth Engine Layers can be used in derivative products through the distributed power of Google Earth Engine.   Google Earth Engine brings together the world's satellite imagery — trillions of scientific measurements dating back over 40 years — and makes it available online with tools for scientists, independent researchers, and nations to mine this massive warehouse of data to detect changes, map trends and quantify differences on the Earth's surface. Applications include: detecting deforestation, classifying land cover, estimating forest biomass and carbon, and mapping the world’s roadless areas.

Croplands Website Purpose: The web application provides a data repository for the team and enables interaction not possible through the USGS website system. Status: Currently in internal testing and use by the team.

Croplands Website: Services, Platforms and Libraries Used

Croplands Website: Objectives Crowdsourcing Training Data: Allow experts and non- experts to contribute local knowledge to training and validation data through a simple interface Interactive Maps: Interactively display multiple data products with easy to understand metadata and legend Data Storage: Provide a standardized data storage solution to serve numerous. Real-Time Distributed Spatial Computing: Enable real- time distributed computing operations through Google Earth Engine

Croplands Website: Crowdsourcing Reference Data: Creating a mechanism for others to contribute their own knowledge on a location to the project. Team Use: Allows our team to review and modify ground data using high resolution satellite imagery, NDVI time series and other data sources as available.

Croplands Website: Interactive Map Layer Legend: Easy to understand layer classes Opacity Controls: Controls allow users to view data products over high resolution imagery. Future: Easily expandable for new data products or additional background layers.

Croplands Website: Data Storage Standardized Data Format: Uses REST web architectural style Restricted Access: Multiple Layers of Access according to Role Backup: Database is backed up on a daily basis and API level data is cached every ten minutes. Capacity: Tested with over 500,000 data entries PublicAPIServerDatabase

Croplands Website: Computing Task Queue: Distributed tasks to send digest s of data changes, process reference photos, find data anomalies or complete other parallel tasks. Real-Time Spatial Calculations: Ability to do computations on spatial data using server-side libraries and interface with Google Earth Engine.  NDVI Time Series  Point in Polygon  Generate Product on Periodic Basis

Data Models Questions still remain but a more flexible structure is being developed. Is a secondary crop from a mixed pixel or a later season?

Locations  Parent object for records, photos, etc.  Point based, but may also be represented by a polygon.  NDVI time series pulled as requested from Google Earth Engine.

Records  Each location may have many records. Currently a unique constraint of a single record per month in a year.  Does not contain spatial data.  Includes land cover type, primary and secondary crop type, water source, and intensity.  May be expanded to include crop health.  Each record has a flag for training or validation.

Common Interface  An Application Programming Interface(API) can allow many individuals to manipulate data in a standardized form.  FAO employees and volunteers may want to access the data from a custom application.  A mobile can consume and manipulate the data in the same way as a web application using the same endpoints.

API Security  JSON Web Tokens: A digitally signed token is generated for a user with a specific expiration time in seconds.  The token is attached to all data requests and validated by the server.  Currently limited to authorizing specific users.

API Permissions TaskPublicUserPartnerTeamAdmin Records Createxxxxx Review xxxx Update xx Delete x Downloads Publicxxxxx Private xxx

# Todo  Get SSL certificates for server endpoints that handle user data.  Rebuild some Angular Directives and Controllers.  Add additional unit tests for frontend application.  Implement timed release for team collected data.  Move source code to public repository for anyone to contribute to.  Move all static content to Amazon Cloudfront and point www domain to Amazon.

# Todo (Continued)  Separation of client application and server application.  Finalize API and data models for rating mechanism.  Input existing global data into system.  Build CSV file import into web application.  Allow filtering of locations by country and continent. Get appropriate libraries running on server.

# Todo (Continued)  NDVI time series aggregation statistics for records generated in browser as requested by user.  Allow selection of records by specific NDVI time series parameters. Find records similar to an existing record.  Simple mobile application for field data collection.  Automatic creation of CSV files saved to GFSAD drive.

#Todo (Continued)  Allow user to overlay kml, geojson or other wms layers on map.  Find alternative to Google Map Engine which is being discontinued 1/16.

Other Ideas  Very simple application allowing user to sketch polygon(field) for a location.  Even simpler application asking ‘Is this cropland?’ with two buttons to help refine the data.  Querying Google Maps for street view data.  Using Google Earth Engine to determine pixel validity.

Mobile Application  What is best approach for converting point data with photo and bearing outside of location to the correct pixel?  Manipulation within mobile application by field personnel.  Manipulation within web application by team members  Auto suggest best location with spatial segmentation and analysis.  Mobile application data should be placed in staging area until approved by team member.