A prototypical tool to discover architecture changes based on multiple monitoring data sources for a distributed system Patrick Schäfer, 08.11.2017, Munich Advisor: Martin Kleehaus

Outline Motivation Research Questions Prototypical Implementation Modeled & Runtime Architecture Problem Statement Research Questions Prototypical Implementation Operating Environment Discovery Concept Implementation Live Demonstration Limitations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Motivation Modeled Enterprise Architecture Business Layer S 4 S 8 S 1 S 3 S 6 S 9 S 2 S 5 S 7 Application Layer VM VM VM VM Technology Layer 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Motivation Runtime architecture Necessity of knowledge about runtime architecture Detecting deviations from modeled architecture Compliance violations / SLA violations Semi-automated root cause analysis Semi-automated Impact analysis State of the art solutions Application Runtime Architecture Discovery A1 A2 V A3 A4 A5 Business Layer S 1 S 4 S 2 S 5 S 6 S 3 S 7 S 9 S 8 Application Layer Technology Layer S 1 S 4 S 2 S 5 S 6 S 3 S 7 S 9 S 8 Discovered Applications Architecture discovery VM Discovered Technology Layer 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Motivation Problem statement Current discovery solutions don‘t own knowledge about the entire architecture including dependencies between and within the architectual layers Consequences comparison of modeled and runtime architecture not possible across all layers root cause analysis is challenging and time consuming Incomplete impact analysis Objective a system, that owns knowledge about all realtime architecture components as well as their inter- and intralayer dependencies S 1 S 4 S 2 S 5 S 6 S 3 S 7 S 9 S 8 Discovered Application Layer Discovered Technology Layer ? A1 A2 V A3 A4 A5 Business Layer 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Research questions 1 How to discover the Microservice Architecture by applying distributed tracing? What are component relationship types and how to discover them automatically? How to discover concurrency and synchronization? How to recognize changes in the Microservice Architecture? How to provide a smart user interface for adding business semantic on top of business services? 2 3 4 5 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Operating Environment Required components of the prototypes operating environment: Instrumented Services Service Repository Eureka Zipkin Server Zipkin Server Service Service Service API Gateway Client Service Service Service Eureka Repository Instrumented Distributed System Legend: Required Component Instrumented Service Optional Component Distributed Tracing Spans 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model Business Layer Process Processes Activities Activity 1 Activity 2 Application Layer Services Service 1 Service 2 Service-Instances Instance 1.1 Instance 1.2 Instance 2 Technology Layer Hardware Hardware 1 Hardware 2 entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Implicit meaning of relations Business Layer Process Processes owns owns Activities Activity 1 is followed by Activity 2 is executed by is executed by Application Layer uses Services Service 1 Service 2 is composed of is composed of is composed of Service-Instances Instance 1.1 Instance 1.2 Instance 2 runs on runs on Technology Layer Closer look relations – every combination of component types => different implicit meaning Hardware Hardware 1 Hardware 2 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Relation classification: Interlayer- and Intralayer-Relations Business Layer Process Processes Activities Activity 1 Activity 2 Application Layer Services Service 1 Service 2 Service-Instances Instance 1.1 Instance 1.2 Instance 2 Technology Layer Hardware Hardware 1 Hardware 2 Hierarchical Interlayer-Relation Non-Hirarchical Intralayer-Relation 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Relation classification: Direct and Indirect Relations Business Layer Process Processes Activities Activity 1 Activity 2 Application Layer Services Service 1 Service 2 Service-Instances Instance 1.1 Instance 1.2 Instance 2 Technology Layer Hardware Hardware 1 Hardware 2 Direct Relation Indirect Relation 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Relation classification: Direct and Indirect Relations Business Layer Process Processes Activities Activity 1 Activity 2 Application Layer Services Service 1 Service 2 Service-Instances Instance 1.1 Instance 1.2 Instance 2 Technology Layer Hardware Hardware 1 Hardware 2 Indirect Relation Direct Relation Path of Indirect Relation 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Indirect Relations are necessary Activity 1 Activity 2 API Gateway Service Service 1 Service 2 Service 3 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Indirect Relations are necessary Activity 1 Activity 2 API Gateway Service Service 1 Service 2 Service 3 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Business Layer Modelling via Web UI: Process and Activity entities Process-flow Added to the architecture model via REST API Processes Activities Services Instances Hardware entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Application / Technology Layer Detection of Services, Instances, Hardware Interlayer-Relations of these types In Realtime by analysing Distributed Tracing Detects new entities only Cyclic by processing Eureka API Detects new, updated or removed entitites Processes Activities Services Instances Hardware entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Service intralayer Relations Detection of relations between Services in Realtime by analysing Distributed Tracing Spans Processes Activities Services Instances Hardware entities of output architecture entities are: Components, Revisions and Relations Distributed Tracing Span production* *Source: http://opentracing.io/documentation/pages/instrumentation/instrumenting-large-systems.html 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Business / Application Relations Detection of Interlayer-Relations between Activities and Services in Realtime Set of regular expressions mapping to activities Set defined via Web UI and REST API Distributed Tracing Spans applied to ruleset Processes Activities Services Instances Hardware entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Implementation: System Components Microservice Architecture Prototype Business Layer Modelling Webapplication Relation Ruleset Architecture- discovery service REST API Eureka Repository Architecture Visualization Distributed Tracing Websocket Bridge Kafka Stream entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Implementation: Environment Zipkin Server replaced by Prototype Prototype extends Zipkin implementation Zipkin Server Features stay complete and unmodified Zipkin Server Service Service Service API Gateway Service Client Service Service entities of output architecture entities are: Components, Revisions and Relations Eureka Repository Instrumented Distributed System 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Implementation: Environment Zipkin Server replaced by Prototype Prototype extends Zipkin implementation Zipkin Server Features stay complete and unmodified Prototype and Zipkin Server Service Service Service API Gateway Service Client Service Service entities of output architecture entities are: Components, Revisions and Relations Eureka Repository Instrumented Distributed System 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Business Process Start Login Select Provider Search Route Book Route Logout Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Accounting Core Service Login Ende Start Select Provider Search Route Book Route Logout DeutscheBahn Mobility Service Zuul API Proxy Travel- companion Service Maps Helper Service DriveNow Mobility Service Business Core Service 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Accounting Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Business Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Business Core Service Start DeutscheBahn Mobility Service Login Select Provider Zuul API Proxy Travel- companion Service Maps Helper Service Search Route Book Route DriveNow Mobility Service Logout Accounting Core Service Ende 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture - Deployment Eureka Zuul API Proxy Travel- companion Service Travel- companion Service DeutscheBahn Mobility Service DeutscheBahn Mobility Service Config Service Business Core Service DriveNow Mobility Service DriveNow Mobility Service Maps Helper Service Maps Helper Service Prototype Accounting Core Service Host 0 (10.0.2.100) Host 1 (10.0.2.110) Host 2.1 (10.0.2.121) Host 2.2 (10.0.2.121) 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Limitations Requires Zipkin-compatible instrumentation as well as Eureka Amount of metainformation depends on instrumentation Raw communications like Database-Access is not detected Virtual Hosts are not detected Business layer is modelled 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

B.Sc. Patrick Schäfer patrick.schaefer@tum.de

Backup 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Live Demonstration Scenario: Microservice Architecture Zuul API Proxy Travel- companion Service Business Core Service DriveNow Mobility Service DeutscheBahn Mobility Service Accounting Core Service Maps Helper Service Synchronous communication Asynchronous communication 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Discovery Concept: Architecture Model - Relations Relation classification: Direct and Indirect Relations Business Layer Process Processes Activities Activity 1 Activity 2 Application Layer Services Service 1 Service 2 Service-Instances Instance 1.1 Instanz 1.2 Instanz 2 Technology Layer Hardware Hardware 1 Hardware 2 Indirect Relation Direct Relation Path of Indirect Relation 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Implementation: Zipkin Server Extension Collektor MySQL Storage Prototypical Implementation Implementation: Zipkin Server Extension Span Span Span Span REST Collektor Kafka Collektor … Architecturediscoveryservice REST API MySQL DB Zipkin UI entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

Prototypical Implementation Implementation: Zipkin Server Extension Collektor Architecture Discovery Storage (Storage-Proxy) Prototypical Implementation Implementation: Zipkin Server Extension Span Span Span Span REST Collektor Kafka Collektor … Architecturediscoveryservice MySQL Storage Architecturediscoveryalgorithm Datamodel REST API MySQL DB Zipkin UI Webapplication Stream REST API MySQL DB entities of output architecture entities are: Components, Revisions and Relations 171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

171108 Schäfer, Patrick - Master's Thesis final presentation © sebis

171108 Schäfer, Patrick - Master's Thesis final presentation © sebis