Presentation is loading. Please wait.

Presentation is loading. Please wait.

Benchmarking Suite: Update

Published byJessica Montgomery Modified over 6 years ago

Similar presentations

Presentation on theme: "Benchmarking Suite: Update"— Presentation transcript:

1

2 Benchmarking Suite: Update
C. Cordeiro, D. Giordano CERN-IT pre-GDB Benchmarking February 2017

3 This is not a performance talk

4 Performance is a key criterion in the design, procurement, and use of computer systems […] to get the highest performance for a given cost. “ ” [1] Performance Measurement is essential Deal with the intrinsic variability and inhomogeneity Compare the presumed and perceived performance Identify performance issues Standard procedure during the procurement process [1] Art of Computer Systems Performance Analysis Techniques For Experimental Design Measurements Simulation And Modeling by Raj Jain , Wiley Computer Publishing, John Wiley & Sons, Inc pre-GDB Benchmarking

5 ease systematic data collection
Automate execution & ease systematic data collection 1 package of benchmark X 1 package of benchmark YZ Pre-heat the environment variables. 1 bag of dependencies Bake benchmark X. After, let It cool. Then bake YZ. Save data for later Benchmark X + Benchmark XY Data collection Data visualization Price: 1-click pre-GDB Benchmarking

6 Benchmarking Model {data collection} * a toolkit done for internal usage. Adoption also by stakeholders of the HEPiX Benchmark working group pre-GDB Benchmarking

7 Benchmark Suite Configurable sequence of Benchmarks Run in any IaaS
Automate and generalize benchmark execution Common output structure Publish results pre-GDB Benchmarking

8 This is a toolkit, not a service

9 (dedicated CentOS6 containers)
NODE $ kv $ DB12 $ whetstone $ hyper-benchmark container mode default mode (dedicated CentOS6 containers) run kv run DB12 run whetstone {} results.json =============================================== RESULTS OF THE OFFLINE BENCHMARK FOR CLOUD Machine classification: i7_1_f6m26s3_mhz Whetstone Benchmark: …… offline publish Benchmarking Suite How it works Benchmark Suite pre-GDB Benchmarking

10 pre-GDB Benchmarking Usage: $0 [OPTIONS] OPTIONS: -q
Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

11 Quiet mode. Do not prompt user -o Offline mode. Do not publish results
Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

12 Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

13 Number of concurrent processes (usually cores) to run
Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

14 Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

15 Unique identifier for the host running this script
Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

16 --queue_host=<hostname>
Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

17 Any additional free text to add to the output JSON
Usage: $0 [OPTIONS] OPTIONS: -q Quiet mode. Do not prompt user -o Offline mode. Do not publish results -i Solves/checks the general and unique dependencies for the specified --benchmarks. If not used, assumes all the dependencies are already installed and configured. NOTE: should run as root --benchmarks=<bmk1;bmk2> Semi-colon separated list of benchmarks to run. Available benchmarks are: - kv - whetstone - DB12 - hyper-benchmark (*) --mp_num=# Number of concurrent processes (usually cores) to run --kv_xml=<xmlFile> Input file for the KV benchmark. If not provided, SingleMuonGenerator is default --uid=<id> Unique identifier for the host running this script --public_ip=<ip> Public IP address of the host running this script --cloud=<cloudName> Cloud name to identify the results - if not specified, CLOUD=test --vo=<VO> Name of the VO responsible for the underlying resource --pnode=<physicalNode> (Optional) Name of the hypervisor machine hosting the VM --queue_port=<portNumber> Port number of the ActiveMQ broker where to send the benchmarking results --queue_host=<hostname> Hostname with the ActiveMQ broker where to send the benchmarking results --username=<username> Username to access the ActiveMQ broker where to send the benchmarking results --password=<password> User password to access ActiveMQ broker where to send the benchmarking results --amq_key=<path_to_key> Key file for the AMQ authentication, without passphrase. Expects --amq_cert --amq_cert=<path_to_cert> Certificate for the AMQ authentication. Expects --amq_key --topic=<topicName> Topic (or Queue) name used in the ActiveMQ broker --freetext=<string> Any additional free text to add to the output JSON pre-GDB Benchmarking

18 Benchmarking Suite Running examples* top tips
export BMK_ROOTDIR=your_local_dir running Whetstone quickly, offline export BMK_LOGDIR=your_log_dir running ATLAS Kit Validation, DIRAC Benchmark and Whetstone, and publish to AMQ with user:pass authentication pushing to AMQ using certificate authentication * for more example please read the documentation at pre-GDB Benchmarking

19 Benchmarking Suite in Action: examples
For direct providers’ performance comparison (addressed by D. Giordano in the previous talk) pre-GDB Benchmarking

20 Benchmarking Suite in Action: examples
For scrutiny analysis of new hardware { "_index":"sdccloud-int", "_type":"vmspec", "_id":"the240s-901bfaac-d a-b943-0a4accf56d24_7b9cc47e-6b51-48cd-94c2e d941_ T15:05:02Z", "_score":null, "_source":{ "message":{ "_timestamp":" T15:05:02Z", "profiles":{ "fastBmk":{ "value":" " } }, "metadata":{ "benchmark_target":"machine", "meminfo": , "UID":"the240s-901bfaac-d a-b943-0a4accf56d24_7b9cc47e-6b51-48cd-94c2-e d941", "classification":"i6_1_f6m63s2_mhz ", "freetext":"VM-1", "cpuname":"Intel(R) Xeon(R) CPU E GHz", "ip":" ", "osdist":"Scientific Linux CERN SLC release 6.8 (Carbon)", "bogomips":4788.9, "VO":null, "cpunum":1, "pnode":"P G07182", "mp_num":"1", "pyver":"2.6.6", "cloud":"CERN-wig_project_011” (addressed in the previous talks) pre-GDB Benchmarking

21 suite benchmarking biography 4M 16+ 125 16 >26k
benchmarks collected since August 6th :57:49 16k benchmarks collected from BOINC 16+ data centers (4 grid sites and over 12 clouds) 125 different CPU architectures Pilot users. Testing and providing the code at /cvmfs/atlas.cern.ch/repo/benchmarks/cern/current 16 different releases from 5 different OS >26k unique IPs Benchmark Working Group Steering, testing and coordinating the activity pre-GDB Benchmarking

22 Highlights Additional benchmarks can be easily plugged in
Same for publishing support for other transportation layers Need to refine packaging and distribution strategy “Grid” adopters prefer to find it in a common CVMFS area KV needs atlas.cern.ch though… Prototyped dependency-free and isolated benchmarks with containers Wider toolkit adoption  more manpower Need support from HEPiX community... pre-GDB Benchmarking

23 Temporary Brochure Documentation Code HEPiX Benchmarking Working Group
Code HEPiX Benchmarking Working Group pre-GDB Benchmarking

24

Download ppt "Benchmarking Suite: Update"

Similar presentations

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
© 2010 VMware Inc. All rights reserved VMware ESX and ESXi Module 3.

© 2010 VMware Inc. All rights reserved VMware ESX and ESXi Module 3.
APACHE SERVER By Innovationframes.com »

APACHE SERVER By Innovationframes.com »
Test results Test definition (1) Istituto Nazionale di Fisica Nucleare, Sezione di Roma; (2) Istituto Nazionale di Fisica Nucleare, Sezione di Bologna.

Test results Test definition (1) Istituto Nazionale di Fisica Nucleare, Sezione di Roma; (2) Istituto Nazionale di Fisica Nucleare, Sezione di Bologna.
DIRAC API DIRAC Project. Overview  DIRAC API  Why APIs are important?  Why advanced users prefer APIs?  How it is done?  What is local mode what.

DIRAC API DIRAC Project. Overview  DIRAC API  Why APIs are important?  Why advanced users prefer APIs?  How it is done?  What is local mode what.
How to Download and Install a Sharp Print Driver on a Mac.

How to Download and Install a Sharp Print Driver on a Mac.
LANDesk Management Gateway

LANDesk Management Gateway
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, 17.07.12, JINR, Dubna.

ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
Presented by: Sanketh Beerabbi University of Central Florida COP6087 - Cloud Computing.

Presented by: Sanketh Beerabbi University of Central Florida COP Cloud Computing.
YAN, Tian On behalf of distributed computing group Institute of High Energy Physics (IHEP), CAS, China CHEP-2015, Apr. 13-17th, OIST, Okinawa.

YAN, Tian On behalf of distributed computing group Institute of High Energy Physics (IHEP), CAS, China CHEP-2015, Apr th, OIST, Okinawa.
F. Brasolin / A. De Salvo – The ATLAS benchmark suite – May,8 2008 Benchmarking ATLAS applications Franco Brasolin - INFN Bologna - Alessandro.

F. Brasolin / A. De Salvo – The ATLAS benchmark suite – May, Benchmarking ATLAS applications Franco Brasolin - INFN Bologna - Alessandro.
QuikTrac 5.5, a validated Motorola Software Solution, allows you to take your Host ERP screens and extend them out to fixed or mobile devices including.

QuikTrac 5.5, a validated Motorola Software Solution, allows you to take your Host ERP screens and extend them out to fixed or mobile devices including.
Advanced Topics StratusLab Tutorial (Orsay, France) 28 November 2012.

Advanced Topics StratusLab Tutorial (Orsay, France) 28 November 2012.
© Chinese University, CSE Dept. Distributed Systems / 9 - 1 Simple Example Open Microsoft Visual Studio 2005:

© Chinese University, CSE Dept. Distributed Systems / Simple Example Open Microsoft Visual Studio 2005:
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1 MSE Virtual Appliance Presenter Name: Patrick Nicholson.

Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1 MSE Virtual Appliance Presenter Name: Patrick Nicholson.
Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Usage of virtualization in gLite certification Andreas Unterkircher.

Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Usage of virtualization in gLite certification Andreas Unterkircher.
EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.
2007.03.20RT-LAB Electrical Applications 1 Opal-RT Technologies Use of the “Store Embedded” mode Solution RT-LAB for PC-104.

RT-LAB Electrical Applications 1 Opal-RT Technologies Use of the “Store Embedded” mode Solution RT-LAB for PC-104.
Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Tools and techniques for managing virtual machine images Andreas.

Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Tools and techniques for managing virtual machine images Andreas.
Database authentication in CORAL and COOL Database authentication in CORAL and COOL Giacomo Govi Giacomo Govi CERN IT/PSS CERN IT/PSS On behalf of the.

Database authentication in CORAL and COOL Database authentication in CORAL and COOL Giacomo Govi Giacomo Govi CERN IT/PSS CERN IT/PSS On behalf of the.

Similar presentations

Ads by Google