1. Survey of Emerging IT Trends and Technologies Chaitan Baru Monday, 10 th Aug 1

2. OUTLINE Trends in data sharing –And, Discovery/Search Trends in service-oriented architectures Trends in computing and data infrastructure The road ahead 2

3. Geoinformatics Use Cases “…a use has access from a terminal to vast stores of data of almost any kind, with the easy ability to visualize, analyze and model those data.” “For a given region (i.e. lat/long extent, plus depth), return a 3D structural model with accompanying geophysical parameters and geologic information, at a specified resolution” 3

4. Implied IT Requirements Search and discovery of resources Integration of heterogeneous 3D / 4D Earth Science data Integration of data with tools Analysis and Visualization –Ability to feed data to tools, and analyze & visualize model outputs (data-centric view…) 4

5. Search and Discovery Searching “structured data”, i.e. metadata catalogs 5 Search Structured metadata catalogs

6. Search and Discovery Searching “unstructured data”, i.e. the Web 6 Search Structured databases are a major component of the “Deep Web” The Web

7. Combined Search and Discovery 7 Search The Web Structured metadata catalogs

8. Advanced Search Proposed: –Geoscience Knowledge System, GeoKnowSys –Built using Yahoo Build Your Own Search (BOSS) service E.g. See wolframalpha.com 8

9. Advanced Search: PaleoLit Research project at Dept of CS, CMU –Dr. Judith Gelernter and Prof. Jamie Carbonell Use ontologies to match search requests to related publications Demo… 9

10. Informatics Issues: The Informatics Progression ITCyber Infrastru cture Cyber Informatics Core Informatics Science Informatics, aka Xinformatics Science, SBAs Informatics Courtesy: Prof. Peter Fox, RPI, CSIG’08

11. The Computer Science / Domain Science continuum 11 Computer  IT  Geoinformatics  Domain  Domain Science Standards Standards Standards Science Topics e.g. Database e.g. ODBC, e.g. Ontologies, e.g. domain e.g. geology Systems, XML GeoSciML vocabularies Semistructure data definitions (Geologic Time, rock description,…)

12. The data interoperability onion 12 Social Networks Semantics Syntax Systems Social Networks Semantics Syntax Systems System Interop –Approaches: e.g., ODBC, JDBC, Java, Web services, … –Purview of: Computer Science Syntactic –Approaches: Schema standards –Purview of: Standards organizations, domain science repositories, data archives Semantic –Approaches: Controlled vocabularies, thesaurii, domain ontologies –Purview of: Domain scientists Social Networks –Approaches: recommendation systems –Purview of: social networking software (CS and domain science, data driven)

13. Software interoperability onion 13 Social Networks Semantics Syntax Systems System Interop –Approaches: e.g., REST, Web services Syntactic –Approaches: e.g., SOAP, WSDL Semantic –Approaches: Controlled vocabularies, thesaurii, domain ontologies –Purview of: Domain scientists Social Networks –Approaches: recommendation systems –Purview of: social networking software Service orchestration via worflow systems

14. Geologic Map Integration

15. Data Mediation Dealing with heterogeneities in (distributed) data sources –Data may be in different “administrative domains”  Manage authentication –Data schemas may be different among sources –Terminologies may be different among sources –Terminologies may be different among sources and user –Software infrastructure (“stack”) may be different Solve the problem with “middleware” –Layers of software between the original application and the end user Mediator –Middleware that bridges across heterogeneities without requiring sources to change

16. AZ NM CO UT NV ID MT WY Shapefile (ESRI) PostGIS Oracle WindowsLinuxiMac DB2 SRB GM L Operating system File storage Database schemas Data Semantics Heterogeneities A Data Integration Example: Geologic Maps

17. FORMATION UNIT_NAME ROCK_TYPE ERA SYSTEM SERIES LITH ROCK_TYPE PERIOD AZ NM CO UT NV ID MT WY WMS Integrated presentation Uniform syntactical structure Uniform spatial definition Advantages Each resource may use a different schema Difficult to build a a uniform query interface for multiple resources. Problems Adopting WMS/WFS: Can provide Syntactic Integration

18. GeoSciML: Can Provide Schema Integration AZ NM CO UT NV ID MT WY GeoSciML Integrated schema Partial integrated semantics Advantages Each resource may use different vocabulary and semantic model. Problem British Rock Classification Multi-hierarchical Rock Classification

19. Semantic Mediation with GeoSciML NMCO British Rock Classification Multi-hierarchical Rock Classification GeoSciML Application Ontology Semantic Mapping Mappings may also be needed between the data and the application ontology E.g., say, mapping 240 mya to Mesozoic

20. Query Rewriting: Example: A Rock Classification Ontology Composition Genesis Fabric Texture

21. Query: Concept Expansion Composition Concept expansion: what else to look for when what else to look for when user asks for ‘Mafic’ user asks for ‘Mafic’

22. Query: Concept Generalization Composition Generalization: finding data that are ‘like’ X and Y finding data that are ‘like’ X and Y

23. Ontology-based Geologic Map Integration: Implemented in GEON Show formations where AGE = ‘Paleozic’ (without age ontology) Show formations where AGE = ‘Paleozic’ (without age ontology) Show formations where AGE = ‘Paleozic’ (with age ontology) Show formations where AGE = ‘Paleozic’ (with age ontology) +/- a few hundred million years domain knowledge domain knowledge Knowledge representation Geologic Age ONTOLOGY Nevada

24. <odal:NamedIndividuals odal:id="RockSample" odal:database="VTDatabase"> Samples RockTexture RockGeoChemistry ModalData MineralChemistry Images ssID GUI generate to ODAL processor The values in the column ssID of the tables Samples, RockTexture, RockGeoChemistry, ModalData,MineralChemistry and Images represent instances of RockSample ODAL: Ontological Database Annotation Language Create a partial model of ontologies from database ODAL, SOQL, and Data Integration Carts™

25. SOQL: Simple Ontology Query Language Query single or many resources via ontologies (i.e., high level logical views) independent of physical representation (i.e. schemas) RockSampleLocation ValueWithUnit float location hasSiO2 value latlong unit string SELECT X.location.*; FROM RockSample X WHERE X.location.lat > 60 AND X.location.long > 100 AND X.hasSiO2.value < 30 AND X.hasSiO2.unit =‘weightPercetage ’ GUI generate to SOQL processor

26. Issues in sharing data: Primary vs secondary (derived) 26 Collect Data Process and Visualize Share Results Share data Share intermediate results

27. Sources of Data Distributed data collections –By individual PIs –“Informal” sharing, e.g. via social network –“Formal” sharing, e.g. via submission to community data archives / databases Centralized data collections –E.g. via a large project (standardized protocols) –By agencies (internal protocols) Metadata to the rescue –Data description standards –Process description standards (workflows) State Surveys and USGS are major sources 27

28. Major Interoperability Efforts OneGeology.org –International initiative of geological surveys to create dynamic geological map data available via the web. US Geoscience Information Network (US GIN) –Led by Lee Allison, AZGS 28

29. Federating Metadata Catalogs Local vs Community “View” –Individual data providers may choose to “export” a community view Direct access to the source may still provide more “rich” access to data Federated Catalogs –The Geosciences Information Network, GIN approach –Adopt standards for catalog content (ISO) and implementation (CSW) 29

30. Interoperation between GEON and GEO GRID Implement CSW interfaces –Collaboration with the NSF PRAGMA project (Pacific Rim Assembly for Grid Middleware Applications) 600 scenes/day Storage Geogrid Catalog Service Web WMS Server WMS URL SRB GEON Catalog Service Web Adapter WMS Server WMS URL ADN CSW REQUEST RESPONSE CSW Composite Service CSW REQUEST RESPONSE GEON GEO Grid

31. Integration & Visualization of 3D/4D data –Derived 3D volumetric model –Multiple isosurfaces with different transparencies –Slices through the volume –Variable gridding: data typically has lower resolution at greater depths –2D surface data: Topography (“2.5D”) Satellite imagery, street maps, geologic maps, fault lines, and other derived features etc. –Bore hole or well data and point observations. “For a given region (i.e. lat/long extent, plus depth), return a 3D structural model with accompanying physical parameters of density, seismic velocities, geochemistry, and geologic ages, using a cell size of 10km”

32. OpenEarth Framework Goals Geoscience Integration: Data types - topography, imagery, bore hole samples, velocity models from seismic tomography, gravity measurements, simulation results… Data coordinate spaces and dimensionality - 2D and 3D spatial representations and 4D that covers the range of geologic processes (EQ cycle to deep time).

33. OpenEarth Framework Goals Structural Integration: Data formats – shapefiles, NetCDF, GeoTIFF, and other formal and defacto standards. Data models - 2D and 3D geometry to semantically richer models of features and relationships between those features. Data delivery methods & Storage Schemes- local files to database queries, web services (WMS, WFS) and services for new data types (large tomographic volumes, etc.).

34. OEF Philosophy OEF focused on integrating data spanning the geosciences. Open software architecture and corresponding software that can properly access, manipulate and visualize the integrated data. Open source to provide the necessary flexibility for academic research and to provide a flexible test bed for new data models and visualization ideas.

35. OEF Architecture

36. Data Integration Services: –Designed to support rapid visualization of integrated datasets –operations to grid data, resample it at multiple resolutions and subdivide data to better support progressive changes to the display as the user pans and zooms

37. OEF Architecture Visualization Tools: –Run on the user's computer, dynamically query spatial and temporal data from the OEF services –Uses 3D graphics hardware for fast display –Open architecture supports multiple visualization tools authored throughout the community (e.g GEON IDV) –New viz capabilities developed as necessary

38. OEF Visualization

39. The software services stack Example: GEON Pushing down the service interface Compute nodesDisk Storage

40. Compute nodesDisk Storage Software as a Service: At different levels of software SaaS PaaS Software as a Service: SaaS –E.g., Google Apps, Salesforce.com, SAP, … Infrastructure as a Service, IaaS –E.g., Amazon EC2, … Platform as a Service, PaaS IaaS

41. The evolving computational architecture Mainframe computers (institutional computing) Minicomputers (departmental computing) Workstations (laboratory computing) Laptops (personal computing) …back to the future..?? 41

42. Cloud Computing: A meeting of trends Data Volumes Price/performance of computing platforms Capabilities of networking and distributed systems Cost of Ownership Models for system management (autonomic computing)

43. Cloud Computing Origins Cloud computing: Many definitions –Here’s one: Use of remote data centers to manage scalable, reliable, on- demand access to applications Origins –Goes back to the need by Web search engines to inexpensively process all the pages on the Web –Done by creating a grid of datacenters and processing data in parallel across them –Development of a parallel data programming environment by Google: MapReduce Data + cloud computing –what about remote centers for scalable, reliable, on-demand access to data?

44. Cloud Computing A different pricing model –No upfront cost of acquisition. Rent don’t buy. Can access 1000’s of processors / disks –Scalability –“Elastic computing” A different model for dealing with system failures –Retry, loose consistency, …

45. Cloud computing for data Data as a service: what is the abstraction for storage? – Table, Blob, Queue – …?? Describing characteristics of the data –Metadata about storage to specify policies to be applied –Security, reliability, performance, etc Scaling to meet application needs – Large configurations –Dealing with virtualization – New failure models Retry, loose consistency

46. Storage as a Service Amazon S3: An example –Charges for Storage, Data Transfer, and Requests (e.g. PUT, COPY, POST, LIST, GET) Issues –Bandwidth to storage –Quality of Service –Storage Elasticity –Privacy / security Standardization efforts –Storage Networking Industry Assocation (SNIA) Technical Working Group (TWG) on Cloud Storage has just started Important Issues –Metadata for storage –Scaling up to large dataset sizes

47. The two sides of Cloud Computing Large distributed infrastructure – “Everything is in the cloud” – Interesting as a proposition for the IT operations of an enterprise – Cloud companies would like to reach deep into enterprise IT – “Our business is not the entrenched data centers in current large organizations, but the new companies…” Large-scale infrastructure in the Datacenter –Seeding the cloud –Shared-nothing parallelism –Data on the cheap…a la Google

48. The NSF Cluster Exploratory (CluE) Program Google-IBM-NSF Cluster –Well over a thousand processors When fully built out, will comprise approximately 1,600 processors –Terabytes of memory –Hundreds of terabytes of storage Open source software –Linux and Apache Hadoop IBM Tivoli –System management, monitoring and dynamic resource provisioning A platform for “apples-to-apples” comparisons –Can reserve time on nodes for exclusive access

49. Our CluE Project Project (PI: Baru; co-PI: Krishnan) –Performance Evaluation of On-Demand Provisioning Strategies for Data Intensive Applications Investigate hybrid software model –Database system / Hadoop system –Some parts of the application require features provided by a DBMS Transactional capability, full SQL support –Other parts of the application can exploit Hadoop model Very large data sets Data parallel processing Loose consistency models Price / performance is an issue –Including energy costs

50. San Andreas Fault LiDAR Dataset: Data Access Patterns B4 Dataset

51. Experiments “On-demand” database vs Hadoop SQL vs Hadoop Energy consumption as a factor in price/performance Platforms to be used Google-IBM cluster OpenCirrus testbed Triton resource

52. The Road Ahead Advanced search engines –Search structured and unstructured data –Deal with display of heterogeneous results –Show provenance of data Sophisticated tools for 3D and 4D data integration –Combination of “server-side” processing and caching and client-side interaction and visualization Service-oriented architecture –Applications and IT infrastructure available as services –Perhaps some of them in “the Cloud” 52

53. 53

54. Dealing with very large data Either the data can be partitioned into segments and processed in parallel –Shared-nothing parallelism Or not –Shared memory systems

55. Parallel Processing of Large Data D P M PPP P Shared Memory

56. Network Shared Nothing D P M D PPP P MMM M D DD

57. Dataset Partitioning Strategy D PPP P MMM M D DD D M P

58. Data partitioning strategies Round-robin – Equal distribution across nodes by data volume Hash –all data with the same key value go to same node Range –all data within a range of values go to the same node Dataset Partitioning Strategy D PPP P MMM M D DD D M P

59. MapReduce / Hadoop Programming environment for very large scale data processing Managing task executions and data transfers in a shared nothing environment –MapReduce: Infrastructure to support data scatter / gather –Distributed data repository (“file system”) Google File System (GFS) Hadoop Distributed File System (HDFS) –Round-robin partitioning of data MapReduce –Google’s proprietary implementation Hadoop –Apache, open source implementation

60. Hadoop vs database MapReduce execution

61. MapReduce vs Database Database –Partition “base tables” into N partitions –Intermediate data can be “re-partitioned” –Intermediate data can be combined –Well-defined algebra for data manipulation (SQL) MapReduce / Hadoop –Partition input data file into M splits –Intermediate data are re-hashed –Intermediate data can be “combined” –Java programs Cost of dynamic vs static partitioning –Run time costs –Storage costs Optimal partitioning –Query and Workload dependent –How to measure any deviations from the optimal? –When to repartition?

62. USGS Role in Geoinformatics Fundamental: Develop, maintain, make accessible: Long-term national and regional geologic, hydrologic, biologic, and geographic databases Long-term national and regional geologic, hydrologic, biologic, and geographic databases Earth and planetary imagery Earth and planetary imagery Open-source models of the complex natural systems and human interaction with that system Open-source models of the complex natural systems and human interaction with that system Physical collections of earth materials, biologic materials, reference standards, geophysical recordings, paper records. Physical collections of earth materials, biologic materials, reference standards, geophysical recordings, paper records. National geologic, biologic, hydrologic, and geographic monitoring systems National geologic, biologic, hydrologic, and geographic monitoring systems Standards of practice for the geologic, hydrologic, biologic, and geographic sciences Standards of practice for the geologic, hydrologic, biologic, and geographic sciences Source: Presentation by Dr. Linda Gundersen, USGS, at Geoinformatics 2007, San Diego, CA.

63. USGS Role in Geoinfomatics All activities: Data creation, modeling, monitoring, collections, standards etc. Must be done in cooperation and collaboration with the public and governmental, academic, and private sector partners and stakeholders. All activities: Data creation, modeling, monitoring, collections, standards etc. Must be done in cooperation and collaboration with the public and governmental, academic, and private sector partners and stakeholders. A critical USGS role: A critical USGS role: facilitate bringing communities together! Source: Presentation by Dr. Linda Gundersen, USGS, at Geoinformatics 2007, San Diego, CA.

64. Data Collections versus Communities of Practice Geoinformatics must evolve beyond the accumulation of data, models, and standards to become the framework for a community of practice in the natural sciences. Geoinformatics must evolve beyond the accumulation of data, models, and standards to become the framework for a community of practice in the natural sciences. Etienne Wegner and Jean Lave coined the term and developed the learning theory of communities of practice – that we learn not only as individuals but as communities. By engaging in communities of practice we increase our capacity and innovation as well as leverage our support for areas of interest. Etienne Wegner and Jean Lave coined the term and developed the learning theory of communities of practice – that we learn not only as individuals but as communities. By engaging in communities of practice we increase our capacity and innovation as well as leverage our support for areas of interest. Source: Presentation by Dr. Linda Gundersen, USGS, at Geoinformatics 2007, San Diego, CA.

65. Creativity, Learning, and Innovation A community of practice is not merely a community with a common interest. But are practitioners who share experiences and learn from each other. They develop a shared repertoire of resources: experiences, stories, tools, vocabularies, ways of addressing recurring problems. This takes time and sustained interaction. Standards of practice and reference materials will grow out of this. But the critical benefits include: creating and sustaining knowledge, leveraging of resources, and rapid learning and innovation. Source: Presentation by Dr. Linda Gundersen, USGS, at Geoinformatics 2007, San Diego, CA.

66. 1000’s of National and Regional Databases The National Map – topographic, elevation, orthoimagery, transportation hydrography etc. The National Map – topographic, elevation, orthoimagery, transportation hydrography etc. Geospatial One Stop-portal Geospatial One Stop-portal MRDATA – Mineral Resources and Related Data MRDATA – Mineral Resources and Related Data The National Geologic Map Database stnadardized community collection of geologic mapping The National Geologic Map Database stnadardized community collection of geologic mapping National Water Information System - NWISWeb National Water Information System - NWISWeb National Geochemical Survey Database (PLUTO, NURE) National Geochemical Survey Database (PLUTO, NURE) National Geophysical Database (aeromag, gravity, aerorad) National Geophysical Database (aeromag, gravity, aerorad) Earthquake Catalogs Earthquake Catalogs North American Breeding Bird Survey North American Breeding Bird Survey National Vegetation/speciation maps National Vegetation/speciation maps National Oil and Gas Assessment National Oil and Gas Assessment National Coal Quality Inventory National Coal Quality Inventory Source: Presentation by Dr. Linda Gundersen, USGS, at Geoinformatics 2007, San Diego, CA.