Benchmark 2 progress report and future directions Sergey Lyskov

Slides:



Advertisements
Similar presentations
How We Manage SaaS Infrastructure Knowledge Track
Advertisements

Content Overview Virtual Disk Port to Intel platform
Test Case Management and Results Tracking System October 2008 D E L I V E R I N G Q U A L I T Y (Short Version)
Low level CASE: Source Code Management. Source Code Management  Also known as Configuration Management  Source Code Managers are tools that: –Archive.
Server selection Multiple servers Add a server UDN selection Channel selection Time selection Duration selection Channel window Time window Current time.
WDK Driver Test Manager. Outline HCT and the history of driver testing Problems to solve Goals of the WDK Driver Test Manager (DTM) Automated Deployment.
SubVersioN – the new Central Service at DESY by Marian Gawron.
European Organization for Nuclear Research Source Control Management Service (Subversion) Brice Copy, Michel Bornand EN-ICE 13 May 2009.
QA Automation Solution. Solution Architecture Test Management tool CI Tool Automation framework Testing Project BDD Tool Text of test to Testing Project.
Using The WDK For Windows Logo And Signature Testing Craig Rowland Program Manager Windows Driver Kits Microsoft Corporation.
Perforce (Version Control Software). Perforce is an enterprise version management system in which users connect to a shared file repository. Perforce.
Version Control with git. Version Control Version control is a system that records changes to a file or set of files over time so that you can recall.
Apache Spark and the future of big data applications Eric Baldeschwieler.
SmartLog X 3 TEAM Basic SmartLog X 3 TEAM Basic DescoEMIT.com USER STATUS USER EDIT TEST LOG ADMIN TEST MACHINE SCHEDULE INSTALL System Requirements:
Visual Studio 2012 or 2013 VisualStudio.com (or) Github (or) Bitbucket (or more) Azure Cross-Platform Command Tools Setup if you want to play too.
RMG Study Group Session I: Git, Sphinx, webRMG Connie Gao 9/20/
Map Reduce: Simplified Data Processing On Large Clusters Jeffery Dean and Sanjay Ghemawat (Google Inc.) OSDI 2004 (Operating Systems Design and Implementation)
Using the WDK for Windows Logo and Signature Testing Craig Rowland Program Manager Windows Driver Kits Microsoft Corporation.
Software Tools and Processes Training and Discussion October 16, :00-4:30 p.m. Jim Willenbring.
TEAM Basic TotalElectrostatic ManagementAwareness&
Microsoft Exchange 2000 Service Pack 2 Features Mark Barringer Support Professional Enterprise Messaging Support Microsoft Corporation.
Large Scale Parallel Print Service Ivan Deloose – David Foster – Ignacio Reguero CHEP 2000 – 8 February 2000 – Padova (I) Presented by Ivan Deloose -
Contents 1.Introduction, architecture 2.Live demonstration 3.Extensibility.
Continuous Integration and Code Review: how IT can help Alex Lossent – IT/PES – Version Control Systems 29-Sep st Forum1.
How to Run a Scenario In HP LoadRunner >>>>>>>>>>>>>>>>>>>>>>
By: Anuj Sharma. Topics covered:  GIT Introduction  GIT Benefits over different tools  GIT workflow  GIT server creation  How to use GIT for first.
CSC350: Learning Management Systems COMSATS Institute of Information Technology (Virtual Campus)
SKYPIAX, how to add Skype capabilities to FreeSWITCH (and Asterisk) CHICAGO, USA, September 2009.
Trilinos Framework Update Roscoe A. Bartlett James M. Willenbring Michael N. Phenow Trilinos User Group Monday, October 31st, : :30 am Building.
Version Control System Lisa Palathingal 03/04/2015.
Intro to Git presented by Brian K. Vagnini Hosted by.
Status Report on the Validation Framework S. Banerjee, D. Elvira, H. Wenzel, J. Yarba Fermilab 15th Geant4 Collaboration Workshop 10/06/
EGEE is a project funded by the European Union under contract IST GLite Integration Infrastructure Integration Team JRA1.
Continuous Integration Make your development life simpler.
Recent Enhancements to Quality Assurance and Case Management within the Emissions Modeling Framework Alison Eyth, R. Partheepan, Q. He Carolina Environmental.
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
Build and Deployment Process Understand NCI’s DevOps and continuous integration requirements Understand NCI’s build and distribution requirements.
TEAM FOUNDATION VERSION CONTROL AN OVERVIEW AND WALKTHROUGH By: Michael Mallar.
2014 Build & Infrastructure Engineering What It Is and Why You Need It Na’Tosha J. Bard October 9, 2014 #GHC
Introduction to Git - Chirag Dani. Objectives Basics of Git Understanding different “Mindset of Git” Demo - Git with Visual Studio.
IBM Software Group ® Jazz Team Build – Part 1 Overview Jonathan.
Using Git with collaboration, code review, and code management for open source and private projects. & Using Terminal to create, and push commits to repositories.
1 Ivan Marsic Rutgers University LECTURE 2: Software Configuration Management.
DECTRIS Ltd Baden-Daettwil Switzerland Continuous Integration and Automatic Testing for the FLUKA release using Jenkins (and Docker)
Version Control Systems
Architecture Review 10/11/2004
Build and Test system for FairRoot
Agenda:- DevOps Tools Chef Jenkins Puppet Apache Ant Apache Maven Logstash Docker New Relic Gradle Git.
How to Contribute to System Testing and Extract Results
Proposed IT Infrastructure for TOP OS project
Project Center Use Cases Revision 2
Project Center Use Cases
Large Scale Parallel Print Service
Segments Basic Uses: slides minutes
LECTURE 2: Software Configuration Management
Version Control overview
Project Center Use Cases
Trends like agile development and continuous integration speak to the modern enterprise’s need to build software hyper-efficiently Jenkins:  a highly.
Version control, using Git
Project Center Use Cases Revision 3
Version Control Systems
Project Center Use Cases Revision 3
What Is Sharepoint? Mohsen Ashkboos
Discrete Convolution Demo
Open Source Continuous Integration System
5 Azure Services Every .NET Developer Needs to Know
Introduction to PHP.
Version/revision control via git
Mark Quirk Head of Technology Developer & Platform Group
Framework Anil
Presentation transcript:

Benchmark 2 progress report and future directions Sergey Lyskov

Benchmark-1 (Current system): Single platform Single machine Test only one revision at the same time Performance not scalable So we can run only a limited number of tests for each revision to keep testing time reasonable

Run on Multiple platforms Test multiple revisions at the same time Results appear on a web page as soon as they are available Performance scales linearly

New website and GitHub integration

Benchmark-1 tests for each revisions: linux.gcc.build.PyRosetta linux.gcc.build.debug linux.gcc.build.release linux.gcc.unit linux.gcc.build.header linux.gcc.build.levels linux.gcc.score

Benchmark-2 tests for each revisions: linux.clang.build.debug linux.clang.build.release linux.clang.mysql.build.debug linux.clang.build.debuglinux.clang.build.release linux.clang.mysql.build.debug linux.clang.postgres.build.debug linux.clang.unit linux.gcc.build.PyRosetta linux.gcc.build.debug linux.gcc.build.release linux.gcc.integration.debug linux.gcc.mpi.build.debug linux.gcc.static.build.release linux.gcc.unit mac.clang.build.PyRosetta mac.clang.build.debug mac.clang.build.header mac.clang.build.levels mac.clang.build.release mac.clang.integration mac.clang.score mac.clang.unit windows.cl.build.PyRosetta 7→21

Tracking Custom Branches on Test Server: Testing server now track 5 branches We can customize which tests to run for each branch Could be very useful for team of developers who work on same project Julia Koehler and Rebecca Alford (rfalford12/mpframework_devel) wrote: Benchmark2 has been a really great resource for developing the membrane code. Frequently running all of the build, unit and integration tests helps us to keep our development compatible with master. Benchmark-2 is also a great collaboration tool - it helps Julia and I keep a running log of changes, and more easily merge our contributions. rfalford12/mpframework_devel

Observers framework for Regression tests Add ‘observers’ text file to your integration tests dir Add lines with lines: baranch1: 1 branch2: 2... for each branch you want to get notifications. Example: De-couple observer info specification from TestServer code Allow to specify multiple observers for multiple branches If score/integration tests got renamed or copied observers info is preserved

Demo

Future Directions Setup addition testing servers: Linux Mac Windows (?) Cluster test, decouple HPC driver from testing scripts and port existing scientific and profile tests (as soon as our testing cluster upgraded to 64-bit OS) Implement binary search for best-effort tests Port performance benchmark and profile tests to Benchmark-2

Your feedback and suggestions are welcome!

Thank you!

Do we need add more build tests? (MPI+Clang, iCC?) Should we have additional status for Integration tests (ie “script failed”) that represent failure of executable to run instead of signaling regression tests changes? Metrics for Scientific Tests. What kind of reports tools do we want? (simple plots? something more powerful?) Can we manage to have scientific tests results to be non-incremental? (its not clear how to manage scheduling for best-effort tests which require comparison with previous runs to deliver results) Topics for discussion:

Now we can comment on revision: So we can ‘add’ info to accidental commit messages And we can comment on test results and such