An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph

For today’s discussion… InfiniteGraph – Background / History – Architecture – What we are working on Consistency Models Challenges/Requirements of Distibution Data model – Schema vs Schema-less Copyright © InfiniteGraph

Building on Objectivity/DB Derived from Objectivity/DB core – Distributed Object Database – Native core (broad platform support) with C++, Java, C# and Python Bindings – Thousands of deployments, many are 24x7x365 – Markets : VLDB, Data Fusion / Metadata, Complex Object Models, Relationship Analytics Copyright © InfiniteGraph

Relationship Analytics Experience Relationship Analytics – Large part of Objy/DB Government business – Significant growth in the commercial space – Many OEM opportunities – Mostly based on graph theory – Hand coded algorithms Copyright © InfiniteGraph

Needed something more… Objectivity Provides – Powerful distributed object storage – Flexible class based persistence – Enterprise DB features But…. – Lots of custom coding – Complex deployment – No generic algorithm framework Copyright © InfiniteGraph

Enter InfiniteGraph Simple Graph focused API Automated distribution and deployment Mostly configuration driven Java class based persistence Property model support Asynchronous navigation Indexing framework Copyright © InfiniteGraph

Some code… Copyright © InfiniteGraph Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”)); alice.addEdge(new Meeting(“Denver”, “ ”), bob); bob.addEdge(new Call(timestamp), carlos); carlos.addEdge(new Payment( ), charlie); bob.addEdge(new Call(timestamp), charlie); AliceCarlosCharlieBob Meets CallsPays Calls

Basic Architecture Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Blueprints User Apps Objectivity/DB Distributed Database Session / TX Management Placement

What Are We Working On ? Copyright © InfiniteGraph

Targeting Large Graphs Graphs grow quickly – Billions of phone calls / day in US – s, social media events, IP Traffic – Financial transactions Some analytics require navigation of large sections of the graph Must distribute data and go parallel Copyright © InfiniteGraph

Consistency Models Trading off full consistency for performance Relax locking, allow stale reads Gain 100x edge ingest rates Great for “social graph” applications These tend to have relaxed consistency requirements API allows choice per operation Copyright © InfiniteGraph

Pipelining Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Session / TX Management Placement (MDP) Placement (Pipelining) V1V1 V1V1 V2V2 V2V2 V3V3 V3V3 E 12 E 23 Pipeline Manager Staging Containers Pipeline Containers E(1->2) E(3->1) E(2->3) E(2->1) E(2->3) E(3->1) E(1->2) E(3->2) E(1->2) E(2->3) E(3->1) E(2->1) E(2->3) E(3->1) E(3->2) E(1->2)

Distributing Navigation Graph algorithms naturally branch Breaking up the process is relatively simple Orchestrating it is more challenging Copyright © InfiniteGraph Alice CarlosCharlieBob Meets CallsPays DaveEveChuck Calls Lives With Meets

Distributed API Application(s) Partition 1Partition 3Partition 2Partition...n Processor Partitioned Graphs are Ugly Copyright © InfiniteGraph

Making it scale… A distributed data tier supports parallel IO A distributed cache/processor tier for processing “navlets” Ability to deal with remote data reads (fast) Navlets hit processor/cache instances based on consistent hashing of their start vertex Copyright © InfiniteGraph

Flexible Data Models Schema isn’t for everyone – Great for relatively static models – Support for inheritance helps – So does auto-evolution Looking to support document style elements – Support for ad-hoc attributes – Ability to mix both approaches Copyright © InfiniteGraph

Other Projects Blueprints / Gremlin support Native Visualizer and Management Console Built in Qualifiers More Indexing Enhancements Algorithms Large scale samples Copyright © InfiniteGraph

Thank you ! Copyright © InfiniteGraph Twitter