BASMATI Integrated Platform

BASMATI Integrated Platform
Young-Woo Jung, ETRI October 10, Seoul

BASMATI Final Review meeting
Outlines BASMATI Architecture Overall BASMATI Work Flow BASMATI Components (Details) Application Repository (AR) Resource Broker (RB) Knowledge Extractor (KE) Decision Maker (DM) + Revenue Sharing Scheme Application Controller (AC) BASMATI Cloud Provider Manager (ACE) SLA Manager (SLAM) Federation Monitoring (FM) BASMATI Unified Data Management Framework (BUDaMaF) BASMATI Workspace – optional BASMATI Integration Environment 10/10/18 BASMATI Final Review meeting 2

BASMATI Architecture Knowledge Extractor Application Repository
Decision Maker BASMATI Workspace Application Controller Business Logic Federation Monitoring SLA Manager Resource Broker BASMATI Cloud Provider Manager (ACE) Application Management Layer Cloud Management Edge Management BUDaMaF Federation Management Layer 10/10/18 BASMATI Final Review meeting 3

Overall BASMATI Work Flow
Knowledge Extractor BASMATI Workspace BASMATIZE!! Application Repository Decision Maker Application Controller BASMATI USER BASMATI REST APIs BEAM Document Business Logic Resource Broker SLA Manager Federation Monitoring BASMATI Cloud Provider Manager (ACE) Edge Node Cloud Provider 1 Cloud Provider N BUDaMaF Edge Node Edge Node … Edge Node VM 1-1 … VM 1-N VM N-1 … VM N-N 10/10/18 BASMATI Final Review meeting 4

BASMATI Components 10/10/18 BASMATI Final Review meeting 5

Application Repository Overview The Application Repository is the component in charge of storing the BEAM documents inside the Platform. Each application "basmatized" has it own BEAM documents, no documents are shared between the applications. Name Format Purpose ID 128-bit UUID Identifies the instance of an application within the BASMATI platform Application Topology TOSCA Describes the application in terms of components and relationships between them Functional Requirements Describes the functional requirements of each component of the application Template Agreement ws-agreement An agreement between a service consumer and service provider specifies one or more service level objectives Decomposition Document XML Defines the partitions of the applications and the degree of replication for each partition Selection Document Provides a ranking of cloud resources for each partition identified in the decomposition document Deployment Document Identifies the mapping between application partition and resources of the federated environment 10/10/18 BASMATI Final Review meeting 6

Application Repository Overview The Application Repository (AR) stores application related information inside the BASMATI integrated platform minimal information are sent back and forth between components The whole set of documents represents the Basmati Enhanced Application Model Practically the AR is a REST-based repository for the retrieval and manipulation of the documents inside the BEAM The AR provides both a RESTFUL API and a Web Admin Access 10/10/18 BASMATI Final Review meeting 7

Application Repository Relation with Other Components REST interfaces for retrieval and Manipulation of the BEAM resources and application files Each application is identified with an unique app ID to facilitate the retrieval of the BEAM resources 10/10/18 BASMATI Final Review meeting 8

Application Repository Implementation & Evaluation The Application Repository has been developed using the Django REST framework. The database backend uses SQLite Can be easily extended to larger / more expensive database solutions The whole service is inside a Docker container for easy and fast deployment References (Deliverable) D3.5 BASMATI Server- and Client-side Applications Adaptation and Reconfiguration: Design and Specification (Deliverable) D3.6 BASMATI Server- and Client- side Applications Adaptation and Reconfiguration: Software Prototype 10/10/18 BASMATI Final Review meeting 9

Resource Broker Overview Objectives Resource Broker(RB) is a component which offers the list of best resources to be used by the applications. Main functionalities Aiding Decision Maker in placement of (part of) application on federated resources. Considering feedback on past allocations to perform prediction on the behavior of the application on resources. Organizing the index of the cloud resources and retrieving the list of cloud resources available for placement and offloading. 10/10/18 BASMATI Final Review meeting 10

Resource Broker Relation with Other Components Application Repository (AR) giving information (BEAM) about applications. RB can acquire actual requirements of the application BASMATI Cloud Provider Manager(ACE) giving information about federation Providing currently available CSPs and resource types SLA Manager giving information when violations happen. The information is used to control the priorities of each resources Decision Maker Requesting to provide best-fit resources to deploy application Receiving list of best-fit resources 10/10/18 BASMATI Final Review meeting 11

Resource Broker Implementation & Evaluation Implementation Written in Python3 Provides REST API to be called by other components Uses REST API and OCCI Interface to call other components Provides Auto-scaling, Auto Recovery features thanks to Django-REST- Framework and Docker Container Technology Evaluation Time to generate a list of best-fit candidates meet requirements < 10 secs RB can accept multiple requirement conditions like price, hardware, software specification, and region. 10/10/18 BASMATI Final Review meeting 12

Resource Broker Implementation & Evaluation References (Deliverable) D4.3 Multimodal Brokerage and Offloading: Design and Specification (Deliverable) D4.5 Brokerage Platform Dynamic Structure: Design and Specification (Deliverable) D4.6 Brokerage Platform Dynamic Structure: Software Prototype (Paper) Dynamic Resource Selection in Cloud Service Broker (HPCS 2017) (Paper) Runtime Adaptive Resource Selection in Cloud Service Brokerage (SerCo 2017–HPCS) 10/10/18 BASMATI Final Review meeting 13

Knowledge Extractor Overview Objective Analysis and modelling of the users and the applications to identify their requirements towards the federated cloud environment Main functionalities: Application resource demands predictions Distribution of users into Point of Interests Deployment plans evaluation 10/10/18 BASMATI Final Review meeting 14

Knowledge Extractor Relation with Other Components 10/10/18 BASMATI Final Review meeting 15

Knowledge Extractor Implementation & Evaluation VM_ocean_37 Observations 3487 CPU 0.0862 Memory 564 MB Net 1.19KB cpu = * time * observations * previous_cpu * previous_Net 10/10/18 BASMATI Final Review meeting 16

Knowledge Extractor Implementation & Evaluation References (Deliverable) D3.1 Analysis and Modelling of Users and Applications: Design and Specification (Deliverable) D3.2 Analysis and Modelling of Users and Applications: Software Prototype (Deliverable) D3.3 Situational Knowledge Acquisition and Inter-Cloud Service Monitoring: Design and Specification (Paper) User Behavior and Application Modeling in Decentralized Edge Cloud Infrastructures (Economics of Grids, Clouds, Systems, and Services, 2017) 10/10/18 BASMATI Final Review meeting 17

Decision Maker Overview Objectives: Performs decision about application placement, i.e. define a mapping between services composing the application and provider resources To carry out adaptive offloading decisions at runtime that are triggered by SLA violations not resolved directly at the infrastructural level Main functionalities of the decision maker are Coordinate all the components to perform the “basmatize” Multi-objective application placement capability through weighing of heterogeneous criteria Capability of working with incomplete data Application of heuristics for obtaining a solution in acceptable time 10/10/18 BASMATI Final Review meeting 18

Decision Maker Relation with Other Components Completed the description of the Decision Maker with its interaction with other BASMATI components in the following way: 10/10/18 BASMATI Final Review meeting 19

Decision Maker Implementation & Evaluation A multi-objective, multi-site application, optimization algorithm for offloading multi-site, dynamically changing applications using genetic metaheuristsics user and application behavior prediction, socio-economic aspects, emerging situations/contexts during runtime Input Values from knowledge extractor and application requirements Apply Genetic Algorithm Select the Best Fitting Deployment Plan Generate Application Deployment Plan 10/10/18 BASMATI Final Review meeting 20

Decision Maker Implementation & Evaluation Compared the results with a benchmark algorithm within three scenarios The solution generated provides lower values for cost, user-node distance and node-node distance but higher values for CPU and memory D – Algorithmic Solution (Full Weight) C – Algorithmic Solution (Equal Weight) B – Algorithmic Solution (Zero Weight) A – Benchmark Solution References (Deliverable) D4.3 Multimodal Brokerage and Offloading: Design and Specification (Deliverable) D4.4 Multimodal Brokerage and Offloading: Software Prototype (Paper) Dynamic application deployment in federations of clouds and edge resources using a multiobjective optimization AI algorithm (FMEC 2018) (Paper) Dynamic Resource Selection in Cloud Service Broker (HPCS 2017) 10/10/18 BASMATI Final Review meeting 21

Revenue Sharing Scheme Introduction Cloud Service Market : 6 companies > 60% of the market (Gartner 2017). Problems of small cloud providers Resource Limitation Limited geographic presence Discrimination by economies of scale. Ideally, cloud federation could solve the problem Access to increased number of cloud infrastructure resources Economies of scale by aggregating both requests and resources Ensure the quality of service, provide guaranteed availability But, the problem is : How can we incentivize cloud providers to join the federation. Research shows the issue of revenue sharing is a hindering factor Requirement : Revenue Sharing Scheme Generate revenue, at least, not less than in individual operation Ensure fair distribution 10/10/18 BASMATI Final Review meeting 22

Revenue Sharing Scheme Methodology Marginal Contribution of a provider i Estimation of value considers both resource contribution and market share of the providers 10/10/18 BASMATI Final Review meeting 23

Revenue Sharing Scheme Evaluation & Conclusion Hourly revenue stream of providers Resource Utilization Average hourly revenue per unit resource Proposed CBRSS Participatory Revenue Distribution in two approaches Aryal, Ram Govinda, and Jörn Altmann. "Fairness in Revenue Sharing for Stable Cloud Federations." International Conference on the Economics of Grids, Clouds, Systems, and Services. Springer, Cham, 2017. Juan Pablo Romero Coronado, Jörn Altmann, “Model for Incentivizing Cloud Service Federation,” GECON 2017, 14th International Conference on Economics of Grids, Clouds, Systems, and Services, Springer, LNCS, Biarritz, France, September 2017. 10/10/18 BASMATI Final Review meeting 24

Application Controller Overview Objectives Application Controller is a component which manages the lifecycle of applications and scale up/down deployed resources of deployed applications Main functionalities Deploy, undeploy, start and stop applications Scale up and down the deployed resources Redeploy failed applications Keep tracking of deployment status of applications 10/10/18 BASMATI Final Review meeting 25

Application Controller Relation with Other Components Decision Maker (DM) or Workspace Calling REST API to deploy, undeploy, start, stop, scale up/down applications Application Repository (AR) Giving application’s deployment plan Storing deployment status of applications ACE Parser Part of ACE Converting deployment plans to manifest and agreement document BASMATI Cloud Provider Manager (ACE) Creating, deleting, starting, stopping and scaling up/down services* * service = deployed instance of application 10/10/18 BASMATI Final Review meeting 26

Application Controller Implementation & Evaluation Implementation Written in Python3 Provides REST API to be called by other components Exploits REST API and OCCI Interface of other components Provides Auto-scaling, Auto Recovery features thanks to Django-REST- Framework and Docker Container Technology Evaluation Time to deploy an application on federated clouds environment < 200 secs Time to scale up a service < 150 sec References (Deliverable) D5.3 Hybrid Infrastructure Management: Design and Specification (Deliverable) D5.4 Hybrid Infrastructure Management: Software Prototype (Paper) Design of Application Controller for BASMATI platform (2017 Spring KIPS Conference) 10/10/18 BASMATI Final Review meeting 27

BASMATI Cloud Provider Manager (ACE) Overview Objectives Automated Hybrid, Multi Cloud Provisioning and Orchestration Automated Federation Description, Operation and Management Main functionalities The BASMATI Cloud Provider Management provides generic and comprehensive solutions to multiple hybrid cloud service provisioning and orchestration requirements of the Basmati Platform. The BASMATI Federation Management provides tools for the commercial and technical definitions required for the automation of service discovery, provisioning and revenue sharing between Cloud Federation Members. 10/10/18 BASMATI Final Review meeting 28

BASMATI Cloud Provider Manager (ACE) Relation with Other Components
Resource Broker Application Repository Application Controller SLA Manager 10/10/18 BASMATI Final Review meeting 29

BASMATI Cloud Provider Manager (ACE) Implementation & Evaluation BEAM Processor Application Topology Technical Requirements Commercial Requirements Cloud Provider Management EDGE Device Management and Provider Interface SCALR Provider Interface AMAZON ECS Provider Interface Federation Management Federation Member Agreement Federation Deployment Federation Billing 10/10/18 BASMATI Final Review meeting 30

BASMATI Cloud Provider Manager (ACE) Implementation & Evaluation References (Deliverable) D5.3 Hybrid Infrastructure Management: Design and Specification (Deliverable) D5.4 Hybrid Infrastructure Management: Software Prototype 10/10/18 BASMATI Final Review meeting 31

SLA Manager Overview Objectives QoS framework for managing applications running in the BASMATI hybrid infrastructure The SLA Manager module receives customer’s requirements and is responsible for the processing of the Business Value defined behavior response to signaling, received from both the Federation Cloud Deployment and the Federation monitoring, concerning incidents and violations detected during the monitoring of the application Guarantees and SLOs. Main functionalities of the SLAM Providing means to negotiate the applications QoS with the federated CSPs Monitoring the application execution continuously Exposing corrective action in case of non-fulfilment of the required QoS 10/10/18 BASMATI Final Review meeting 32

Publication and Discovery Monitoring and Evaluation
SLA Manager Relation with Other Components Creation Publication and Discovery Negotiation Monitoring and Evaluation Accounting Application Controller Cloud Service Providers Monitoring Optimization of Resource Selection 10/10/18 BASMATI Final Review meeting 33

SLA Manager Implementation&Evaluation 10/10/18 BASMATI Final Review meeting 34

SLA Manager Implementation & Evaluation References (Deliverable) D5.5 Service, security and privacy quality enforcement: Design and Specification (Deliverable) D5.6 Service, security and privacy quality enforcement: Software Prototype 10/10/18 BASMATI Final Review meeting 35

Federation Monitoring
Federation Monitoring Overview Objectives Collecting monitoring data for VMs’ resources. Collecting monitoring data in the federated cloud environment, regardless of any type of OS and any type of cloud service providers. Providing monitoring capability for every managed VMs as well as for every federated CSPs. Providing REST APIs to other components. Main funtionalities Integrated monitoring for federated clouds Supporting a real-time monitoring capability Providing alarm functionality based on SLA policy Generic monitoring framework independent of any CPM Free to extend functionalities Federation Monitoring Monitoring API Module Real-time Analysis Module Alarm Check Module Monitoring Collector Management Module Monitoring Metadata DB Monitoring DB 10/10/18 BASMATI Final Review meeting 36

Federation Monitoring
Federation Monitoring Relation with Other Components The Federation Monitoring interacts with other BASMATI components in the following way: Federation Monitoring ACE Knowledge Extractor SLA Manager CSP/ASP, VM and Application info Monitoring data for CSP, VM and Application BASMATI Workspace Monitoring data for CSP and VM 10/10/18 BASMATI Final Review meeting 37

Federation Monitoring Implementation & Evaluation Implementation Written in Python3 and using Django-REST-framework All sub components of FM have been developed inside a Docker container Providing REST API to be called by other components Evaluation FM collects 28 monitoring items of VM. FM collects monitoring data every 15 seconds. FM provides real-time monitoring (less than 3 seconds) for immediate change detection References (Deliverable) D3.3 Situational Knowledge Acquisition and Inter-Cloud Service Monitoring: Design and Specification (Deliverable) D3.4 BASMATI Service Monitoring Framework 10/10/18 BASMATI Final Review meeting 38

BUDaMaF Overview Objectives Creating a unified method of performing data related tasks inside the BASMATI ecosystem, while preserving QoS and security restrictions Main functionalities Facilitating the storage and trafficking of data through the BASMATI ecosystem Ensuring optimal QoS regarding the data requests Enforcing security and anonymization throughout the federation Providing data store live scaling and migration tools 10/10/18 BASMATI Final Review meeting 39

BUDaMaF Relation with Other Components
BUDaMaF is a toolkit for data management tasks. As such it is used by each hosted application and BASMATI components that create or access databases. It provides amongst others: A module that creates a secure, encrypted data transfer tunnel with no effect on the hosted applications or on the functionality of the machine hosting the application A data store scaling server that can request new cloud resources and scale data stores in real time Hosted Application or BASMATI Component BUDaMaF Core Security Module Replication Data Store management Data management Migration Scaling Access Lock Creation Migration Encrypt and Store Destruction 10/10/18 BASMATI Final Review meeting 40

BUDaMaF Implementation & Evaluation BUDaMaF security module. Installed and run through a pre-created script. Evaluated on Ubuntu V16 VMs as part of the DasFest Trilateration application. Encryption/Decryption in under 1 second for each data transfer. Encryption overhead of up to 30% in worst case scenario. The data and data store management framework Operated through the core Support for mongoDB Partial support for HDFS Evaluated on mongoDB with Ubuntu v16 Virtual Machines (VMs) New mongoDB node with fresh VM in under 3 minutes New mongoDB node with rezerve VM in under 1 minute 10/10/18 BASMATI Final Review meeting 41

BUDaMaF Implementation & Evaluation References (Deliverable) D5.1 Scalable big data management: Design and Specification (Deliverable) D5.2 Scalable big data management: Software Prototype 10/10/18 BASMATI Final Review meeting 42

BASMATI Workspace Overview Objectives BASMATI Workspace provides a web-based graphical user interface(GUI) for administrator and users of BASMATI platform. Main functionalities Supervising all CSPs, user applications and services of BASMATI platform Providing the user interfaces for the life cycle management of user applications Monitoring deployed resources on federated cloud environment 10/10/18 BASMATI Final Review meeting 43

BASMATI Workspace Relation with Other Components Application Repository Creating and deleting applications Editing and viewing BEAM documents and topologies Viewing deployment documents and deployment status Decision Maker Calling “BASMATIZE” command to generate deployment plans Application Controller Calling Deploy, Undeploy, Start, Stop, Scale-up and Scale-down to control application lifecycles Federation Monitoring Visualizing monitoring data BASMATI Cloud Provider Manager (ACE) Getting detailed information about deployed services 10/10/18 BASMATI Final Review meeting 44

BASMATI Workspace Implementation 10/10/18 BASMATI Final Review meeting 45

BASMATI Integration Environment 10/10/18 BASMATI Final Review meeting 46

Thank you! Acknowledgements: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no and from ICT R&D program of Korean Ministry of Science, ICT and Future Planning no. R Visit us: 10/10/18 BASMATI Final Review meeting 47

BASMATI Integrated Platform

Similar presentations

Presentation on theme: "BASMATI Integrated Platform"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

BASMATI Integrated Platform

Similar presentations

Presentation on theme: "BASMATI Integrated Platform"— Presentation transcript:

Similar presentations

About project

Feedback