WP3 System Architecture & System Integration By (Stein) Runar Bergheim Asplan Viak Internet

PATHS Project Review 12th March 2014, Luxembourg Overview of Presentation Formal tasks & deliverables Revisiting the PATHS system architecture What is new in version 2 Challenges encountered and their solutions Summing up

PATHS Project Review 12th March 2014, Luxembourg WP3 Deliverables D3.1: Specification of System Architecture  Due Month 8, August 2011 [ submitted: , +1 month ] D3.2: First Prototype and Documentation  Month 16, April 2012 [ submitted: , +1 month ] D3.3: Second Prototype and Documentation  Month 28, April 2013 [ submitted: , ~roughly on time ]

PATHS Project Review 12th March 2014, Luxembourg Data The PATHS System Revisited PATHS Web API (JSON/XML) User Interfrace Database Solr Index Auth Query / View Edit / Delete Data Loader Cloud WP1,4 Wp2

PATHS Project Review 12th March 2014, Luxembourg WHAT IS NEW IN V2.0 WP3 – System Architecture

PATHS Project Review 12th March 2014, Luxembourg New Features 2nd Prototype Vocabul- aries, thesauri 2nd user interface Mobile application Recomm- endations Visual- ization Geo- location PATHS Web API (JSON/XML) Cloud Existing system 1st user interface Data Loader Dates Types

PATHS Project Review 12th March 2014, Luxembourg Extended Data Model...uses URIs as keys

PATHS Project Review 12th March 2014, Luxembourg Database + Solr Index ~1.88 mill items ~19.9 mill b.ground links ~51.1 mill similarity links ~44k multi- hierarchical topics PostgreSQL Solr

PATHS Project Review 12th March 2014, Luxembourg System Overview Main Web API PostgreSQL, PostGIS, Apache, IIS, Solr, Tomcat, C#.NET Prototype UI HTML5, Javascript, Python Recommender methods Java, Perl, PostgreSQL Sentiment methods Javascript Java JSON

PATHS Project Review 12th March 2014, Luxembourg PATHS API Web Services Usr: Users, auth and behavior Path: Path and node authoring & retrieval Item: ESEPaths interaction, background links, similarity links Topic: topic hierarchies, query Search: query items, paths SolrProxy: access to Solr select end-point

PATHS Project Review 12th March 2014, Luxembourg PATHS API Clients 2nd Prototype  iPad Application  Link/screenshot Branching paths, map based visualization, thesaurus browser Exploration on mobile platform, social media interaction

PATHS Project Review 12th March 2014, Luxembourg CHALLENGES & THEIR SOLUTIONS System Architecture and Integration

PATHS Project Review 12th March 2014, Luxembourg Parallel vs Sequential Parallel execution of dependant WPs  Work Package 2, 3 and 4 have parallel activities but sequential dependencies Impact, solution, lessons learned  Some sub-optimal but necessary technology decisions  WP3 is developed using a sub-set of data  WP4 is developed assuming, not knowning, how WP3 will work  Extra integration work at the end of the process  A lot of valuable knowledge about the optimization of different systems, technologies and service interactions  Summarized in D3.3

PATHS Project Review 12th March 2014, Luxembourg Platform Issues Platform complexity  The platform includes a «scientific» quantity of technologies, various stages of releases: beta, 0.x versions, Platform compatibility  A component designed for Apache with Python on Linux does NOT necessarily work the same way on IIS with Python on Windows – notable at migration Impact, solution, lessons learned  Inconvenience for user testing, field trials  Extra efforts required for intensive debugging  Lessons learned: dependency awareness  Learnings summarized in D3.3, D5.3

PATHS Project Review 12th March 2014, Luxembourg Data API vs local data Virtualized data vs local data layers  The overhead of doing multiple parallel/sequential queries over HTTP are far greater than doing the same towards i.e. a local database or service Challenging the REST architecture  Necessary to make authoring-API session-aware, i.e. authoring is not done through a RESTful service Impact, solution, lessons learned  An application that is going to use an API needs to be aware at design time  Validating alternative technologies, e.g. MongoDB for multi-hierarchies  Valuable knowledge on Web APIs, useful in follow-up of exploitation plan, documented in several of the presentations on slide-share, publications

PATHS Project Review 12th March 2014, Luxembourg WP3: Summing Up Strong points  Enables others to build apps on the same virtual data layer  Demonstrates fully virtualized system architecture with sophisticated app  Modular architecture allows for re- use of all OR parts of the PATHS system  Open standards with interfaces to mainstream of European GLAM institutions  The use of URIs as keys permits ANY data – not only ESE Opportunities ahead  Many - in various stages of follow- up, please refer to presentation on Exploitation Plan

Questions & Thanks for the Attention (Stein) Runar Bergheim