Parametric Bottlenecks Testing Catalogue (POSCA) Yu Yang 20161201

Contents Goals and scope of POSCA testsuite Landscape of POSCA testsuite Interaction with Yardstick for POSCA Metrics & Tools for POSCA Work Plan for POSCA testsuite POSCA in Bottlenecks D release Draft Demo

Bottlenecks Testing Results Parametric Bottlenecks Testing Catalogue in Bottlenecks (POSCA) Test Cases Test Results Network Storage Compute Midware APP Bottlenecks Testing Results DB Yardstick Bottlenecks SFC IPv6 SDNVPN Installers OVS4NFV OPNFV Reference Platform 4. Performance Improvement 2. Feedback bottlenecks 3. Upstream Develop 1. Classified bottlenecks Cperf VSPERF StorPerf

Interaction with Yardstick for POSCA Task.iter() Metric Bottlenecks N S C M A Overall Compute Network Storage Midware APP Aver. Res. Time Max Num. User Throughput Requests/Sec Latency Decoupling Bottlenecks Restful Yardstick Metric Max Value Min. Value Aver. Value Compute Storage Network Midware APP SW Tuning HW Tuning Protocol Opt. SLA … Logics Locate Bottlenecks Report 4

Metrics & Tools Discussion Target Metrics Set for Specific Bottlenecks Feature testing results could be organized into different metrics sets to find the bottlenecks Monitoring Compute: latency, utilization of CPU, cache size, etc. Network: throughput, number of connection, packet delay, etc. Storage: memory available mbytes, pages/sec, idle time, etc. Midware: concurrent request, response speed, throughput, etc. APP: throughput, latency, request/user, etc.

Metrics from Yardstick

https://cloud.google.com/monitoring/api/metrics A Brief List of Metrics and Tools Category Bottlenecks Metrics Set Description M&T List https://cloud.google.com/monitoring/api/metrics http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html http://www.applicationperformancemanagement.org/network-monitoring/network-monitoring-tools/ Compute Short of Processor (System\%Total processor time, Processor %Processor Time, system\Processor Queue Length) Metrics 2 is for SQL Server PPT is to avoid memory shortage SPQL is to trace LB of processors Network latency reponse time Metrics 1 is for web server Metrics 2 is for throughput (reponse time, %package loss) where the network congestion occur and throuput reaches it bottleneck Storage Short of Memory (Memory Available MBytes) (Page Reads/Sec, Page/Sec) PS is not necessarily lack of memory when it is high, maybe an application sequentially reading a memory mapped file (%Disk Time, Page Reads/Sec, Avg.Disk Queue Length) Short of memory will cause using Disk Cache memory leak (Memory Available MBytes) (Process\Private Bytes, Process\Working Set, Memory\Available bytes) PPB an PWS are couters that increase when MAB decreases I/O (PhysicalDisk/%Disk time，PhysicalDisk/%Idle Time，Physical Disk\ Avg.Disk Queue Length， Disk sec/Transfer) Only DT is hight, then Disk is not the bottlenecks. PRS is to avoid memory shortage More are under discussion and planed to develop

Work Plan for D Release Adding testing suite to Bottlenecks projects Jenkins job and proposed test suite Code structure in the Bottlenecks repo Determine metrics set and tools for the initial setup Compute: Short of Processor Network: bandwidth, latency and throughput Storage: Short of Memory, memory leak, I/O Determine test story/test cases Factor test: base test cases that Feature test and Optimization will be dependant on Feature test: test cases for features/scenarios Optimization test: test to tune the system parameter RESTful API Flask + Swagger CLI framework Click Dashboard for test result Kibana & Elasticsearch

Work Plan for D release Proposed Test Suite

POSCA Test Cases Discussion & Current Status Bottlenecks VSTF Rubbos Posca Factor Test Feature Test Optimization Test Test Case Description System Bandwidth Throughput Compute Burdon CPU Load TX/RX Cache Size Cache Size TX/RX Package Size Package Size Protocol Limit Test Case Description

POSCA Demo Jan 2016

Demo Contents Bottlenecks CLI Framework POSCA Demo CLI virtual environment setup POSCA testcase run CLI virtual environment clearup POSCA Demo Docker-compose for POSCA setting up dockers for different components Calling Yardstick to run testcases Bottlenecks for system bandwidth Kibana & Elasticsearch for testing results

Demo Steps Step 1: Build docker images Step 2: Docker-compose setting up SUT Step 3: Login bottlenecks docker and set CLI virtual enviroment Step 4: Run testcase Step 5: Results sent to designed EK Step 6: Clearup CLI virtual enviroment, logout bottlenecks docker and clearup docker resources

Thank You Jan 2016

Some Storage Metrics Capacity utilisation: in terms of percent/GB of space used, as well as subcategories such as raw, formatted, free, allocated or allocated not used I/O per second (IOPS) Bandwidth Latency Access time: read, write, random Energy usage: from macro (subsystem) to micro (device or component) Mean time between failure (MTBF) Mean time to repair or replace (MTTR) failed subsystems/components Recovery point objective (RPO): The point in time to which you want data restored Recovery time objective (RTO): The time period in which data to the point required by the RPO must be restored.