1. Managing Big Data and Little Data with WorldMap + Dataverse
Harvard Datafest
January 18, 2017
Managing Big Data and Little Data with WorldMap + Dataverse
Ben Lewis , Paolo Corti, Raman Prasad, David Smiley

2. Major New Capabilities
WorldMap <-> Dataverse Integration
WorldMap <-> HHypermap (Harvard Hypermap)
BOP (Billion Object Platform)
This is a quick overview. Contact me or CGA for more information on any of these systems!
Just 30 minutes to this will be a quick overview. I'm always available if you have trouble using anything or need more information on anything.

3. 1. WorldMap<->Dataverse Integration
Problem: Need maps visualization in Dataverse
Build API’s enable WorldMap and Dataverse to work together.
Map functions now supported in Dataverse
Table geocoding and visualization by field (census, zip, etc.)
Table geocoding and visualization by lat /long
Visualization of geodata (Shapefiles)
Funded by Boston Area Research Initiative and NSF, developed with the Dataverse team.
First I will talk about work we are now completing together with the Dataverse team funded by BARI and NSF to develop APIs to link Dataverse and the mapping platform worldmap in order to start providing mapping services within Dataverse.

4. 1. Dataverse<->WorldMap Integration
Demo

5. 2. WorldMap <-> HHypermap
Problem: There exist millions of geospatial map services on thousands of servers around the world but no way to easy way to find them and use them.
Build a comprehensive registry of map services and make it available via an API
Build a new kind of visual search interface to make the data usable
Funded by NEH

6. 2. WorldMap <-> HHypermap
Support data search within WorldMap or any system via API.
Demo

7. 3. BOP (Billion Object Platform)
Problem: It is very difficult for researchers to explore large, streaming datasets.
Build a platform to support interactive visualization of billions of spatio-temporal objects
Provide an API to make capabilities broadly accessible
Start with geotweets
Funded by Sloan, developed with Dataverse team

8. 3. BOP (Billion Object Platform)
Provides access to latest billion global geotweets. Could be used to support exploration of any large geodataset.
Demo

9. BOP client with time histogram

10. All code open source on GitHub

11. Thank you

12. Simple BOP Client

13. BOP Search Stack Apache Lucene: text search library
Apache Solr: a search platform built on Lucene
Apache Kafka: a message broker written in Scala for streaming data
Apache ZooKeeper: coordinates Lucene shards
Swagger: a framework for building APIs
scikit-learn library: Machine Learning in Python
OpenLayers: mapping client
AngularJS: a javascript framework for client development

14. Streaming big data