1. Parametric Bottlenecks Testing Catalogue (POSCA)
Yu Yang

2. Contents Goals and scope of POSCA testsuite
Landscape of POSCA testsuite
Interaction with Yardstick for POSCA
Metrics & Tools for POSCA
Work Plan for POSCA testsuite
POSCA in Bottlenecks D release
Draft Demo

3. Bottlenecks Testing Results
Parametric Bottlenecks Testing Catalogue in Bottlenecks (POSCA)
Test Cases
Test Results
Network
Storage
Compute
Midware
APP
Bottlenecks Testing Results DB
Yardstick
Bottlenecks
SFC
IPv6
SDNVPN
Installers
OVS4NFV
OPNFV Reference
Platform
4. Performance Improvement
2. Feedback bottlenecks
3. Upstream Develop
1. Classified bottlenecks
Cperf
VSPERF
StorPerf

4. Interaction with Yardstick for POSCA
Task.iter()
Metric
Bottlenecks N S C M A
Overall
Compute
Network
Storage
Midware
APP
Aver. Res. Time
Max Num. User
Throughput
Requests/Sec
Latency
Decoupling
Bottlenecks
Restful
Yardstick
Metric
Max Value
Min. Value
Aver. Value
Compute
Storage
Network
Midware
APP
SW Tuning
HW Tuning
Protocol Opt.
SLA …
Logics
Locate Bottlenecks
Report 4

5. Metrics & Tools Discussion
Target
Metrics Set for Specific Bottlenecks
Feature testing results could be organized into different metrics sets to find the bottlenecks
Monitoring
Compute: latency, utilization of CPU, cache size, etc.
Network: throughput, number of connection, packet delay, etc.
Storage: memory available mbytes, pages/sec, idle time, etc.
Midware: concurrent request, response speed, throughput, etc.
APP: throughput, latency, request/user, etc.

6. Metrics from Yardstick

7. https://cloud.google.com/monitoring/api/metrics
A Brief List of Metrics and Tools
Category
Bottlenecks
Metrics Set
Description
M&T List
Compute
Short of Processor
(System\%Total processor time, Processor %Processor Time, system\Processor Queue Length)
Metrics 2 is for SQL Server PPT is to avoid memory shortage SPQL is to trace LB of processors
Network
latency
reponse time
Metrics 1 is for web server Metrics 2 is for
throughput
(reponse time, %package loss)
where the network congestion occur and throuput reaches it bottleneck
Storage
Short of Memory (Memory Available MBytes)
(Page Reads/Sec, Page/Sec)
PS is not necessarily lack of memory when it is high, maybe an application sequentially reading a memory mapped file
(%Disk Time, Page Reads/Sec, Avg.Disk Queue Length)
Short of memory will cause using Disk Cache
memory leak (Memory Available MBytes)
(Process\Private Bytes, Process\Working Set, Memory\Available bytes)
PPB an PWS are couters that increase when MAB decreases
I/O
(PhysicalDisk/%Disk time，PhysicalDisk/%Idle Time，Physical Disk\ Avg.Disk Queue Length， Disk sec/Transfer)
Only DT is hight, then Disk is not the bottlenecks. PRS is to avoid memory shortage
More are under discussion and planed to develop

8. Work Plan for D Release Adding testing suite to Bottlenecks projects
Jenkins job and proposed test suite
Code structure in the Bottlenecks repo
Determine metrics set and tools for the initial setup
Compute: Short of Processor
Network: bandwidth, latency and throughput
Storage: Short of Memory, memory leak, I/O
Determine test story/test cases
Factor test: base test cases that Feature test and Optimization will be dependant on
Feature test: test cases for features/scenarios
Optimization test: test to tune the system parameter
RESTful API
Flask + Swagger
CLI framework
Click
Dashboard for test result
Kibana & Elasticsearch

9. Work Plan for D release
Proposed Test Suite

10. POSCA Test Cases Discussion & Current Status
Bottlenecks
VSTF
Rubbos
Posca
Factor Test
Feature Test
Optimization Test
Test Case
Description
System Bandwidth
Throughput
Compute Burdon
CPU Load
TX/RX Cache Size
Cache Size
TX/RX Package Size
Package Size
Protocol
Limit
Test Case
Description

11. POSCA Demo
Jan 2016

12. Demo Contents Bottlenecks CLI Framework POSCA Demo
CLI virtual environment setup
POSCA testcase run
CLI virtual environment clearup
POSCA Demo
Docker-compose for POSCA setting up dockers for different components
Calling Yardstick to run testcases
Bottlenecks for system bandwidth
Kibana & Elasticsearch for testing results

13. Demo Steps Step 1: Build docker images
Step 2: Docker-compose setting up SUT
Step 3: Login bottlenecks docker and set CLI virtual enviroment
Step 4: Run testcase
Step 5: Results sent to designed EK
Step 6: Clearup CLI virtual enviroment, logout bottlenecks docker and clearup docker resources

14. Thank You
Jan 2016

15. Some Storage Metrics
Capacity utilisation: in terms of percent/GB of space used, as well as subcategories such as raw, formatted, free, allocated or allocated not used
I/O per second (IOPS)
Bandwidth
Latency
Access time: read, write, random
Energy usage: from macro (subsystem) to micro (device or component)
Mean time between failure (MTBF)
Mean time to repair or replace (MTTR) failed subsystems/components
Recovery point objective (RPO): The point in time to which you want data restored
Recovery time objective (RTO): The time period in which data to the point required by the RPO must be restored.