Instrumenting CernVM-FS Code José Molina Colmenero CERN PH-SFT 1June 29th 2015

Main tasks so far Improve test coverage – Discover bugs and make sure everything works fine – Presentation of the results Benchmarking – To know the consequences of our changes – Presentation of the results Automatic integration: Jenkins 2June 29th 2015

Test coverage: current situation Mainly focused on developing new unit tests – C++: Currently 626 tests from 59 test cases – Python: 24 tests from 6 test cases (85% covered) – Also integration tests and stress tests check how the entire system works Main target: to reach a 75% test coverage at the end of 2015 Measurement performed using GCC Code Coverage Report and the Gcov toolGcov – Data presented using an external tool: gcovrgcovr – Results can be found herehere 3June 29th 2015

Test coverage: Gcov 4June 29th 2015

Test coverage: Gcov in detail 5June 29th 2015

Test coverage: Gcov global results Current status: If we include all CernVM-FS sources the total coverage is even lower – We expect to find around lines and we have currently tested around 43% – When I arrived the coverage was around 25% 6June 29th 2015

Test coverage: Jenkins Test coverage presentation is already integrated in Jenkins – It allows us to automatically keep the test coverage updated Jenkins project: CvmfsTestCoverage 7June 29th 2015

Test coverage: Jenkins It would be necessary to install a new plug-in to publish the test coverage XML report – Cobertura plug-in Cobertura plug-in – It would offer a visualization of the test coverage evolution HTML results with per-line details will be uploaded to a server – Check it out herehere 8June 29th 2015

Benchmarking: what for? It is important not only to know if what you do “apparently” does not break anything, but if the performance is good In CernVM-FS there are many elements that rely on others, so that if one of them is changed there could be an important impact in the general performance – How do we measure that? 9June 29th 2015

Benchmarking: initial status In CernVM-FS there is already developed a statistics system that provides diverse counters The graphical visualization can help a lot to spot trends over longer periods It is necessary to execute the program with a realistic run so that we can consider those counters as valid values 10June 29th 2015

Benchmarking: infrastructure Benchmarks for four important experiment frameworks – ATLAS, LHCB, ALICE and CMS at the moment Many different run options – Benchmarks to run, number of iterations, cold/warm cache, number of instances in parallel… – As flexible as possible in order to integrate easily new tools Already integrated in Jenkins – CvmfsBenchmark project CvmfsBenchmark project 11June 29th 2015

Benchmarking: analysis tools First try: IgprofIgprof – We could not make it work – Problems with forks It was necessary to modify the source code – CernVM-FS uses many signals and features that interfere with Igprof – Not meaningful results for a non-CPU-intense software as CernVM-FS due to long idle times Second try: ValgrindValgrind – Two tools at the moment: Callgrind and Memcheck – Necessary to add new starting options to avoid forks simple_option_parsing option Other tools could be easily added if necessary 12June 29th 2015

Benchmarking: Callgrind Callgrind is a Valgrind’s tool that shows the call graph and how much time we spend in each function – Good to determine the functions that should be improved to get a much better performance out of it It totally emulates the code, so the results are 100% accurate – But runs 20~30 times slower That is not a problem for CernVM-FS because is not CPU- intense Tool for a graphical visualization: kcachegrindkcachegrind 13June 29th 2015

Benchmarking: Callgrind output 14June 29th 2015

Benchmarking: Memcheck Memcheck is a Valgrind’s tool that searches for memory problems – Very helpful to identify potential memory bugs that are often hard to detect – But also to improve memory usage finding out memory leaks – However it does not have at the moment a friendly graphical visualization It is possible to give it a try using memcheckviewmemcheckview Integration with Jenkins – Valgrind plug-in that could be installed Valgrind plug-in 15June 29th 2015

Benchmarking: Valgrind tools A considerably big problem is that visualization tools are mostly desktop programs – Inconvenient to visualize the results – However, right now it is the only practical way An online tool would come in handy – No good ones at the moment – We tried with Webgrind, but it is an old project that does not solve all our requirements and has security leaksWebgrind 16June 29th 2015

Benchmarking: statistics CernVM-FS generates a statistics file that includes information about all the counters spread through the code – Also externally observed numbers are added by the benchmarking infrastructure: time spent and memory consumption at the moment A python script parses this file and sends the data to a special databasepython script – Distributed time series database: InfluxDBInfluxDB – SQL-like syntax for queries, but totally different approach for insertions Another third-party software takes care of the graphical representation – Grafana is a highly configurable program used to show online the evolution of our counters Grafana – You can find our instance herehere – New data is automatically sent every day through our Jenkins project 17June 29th 2015

Benchmarking: InfluxDB 18June 29th 2015

Benchmarking: Grafana 19June 29th 2015

Benchmarking: Grafana 20June 29th 2015

Benchmarking: configuring Grafana 21June 29th 2015

Benchmarking: branch comparison Situation: someone submits a pull request with a new feature – It compiles and passes the unit tests, and looks good – But does it have a considerable impact on the general performance? It is complicated to run the benchmarks on both branches – It requires to install the binaries – Generally not a good idea – It is necessary to use a cleaned environment every iteration 22June 29th 2015

Benchmarking: Docker Thanks to Docker it is possible to run the benchmarks on each branch using exactly the same environmentDocker – And it is cleaned up by itself Development of a python script that takes care of initializing the instances and running each benchmark on the concrete branchpython script – Highly configurable External repository and branch Basically the same benchmark parameters can be specified 23June 29th 2015

Benchmarking: Docker June 29th Extracted from

Benchmarking: comparison result The script generates a CSV file with the result 25June 29th 2015

Benchmarking: comparison example Recently we added sha2 functionality – It is supposed to consume more memory – But what does “more” mean? This case has been tested using the previously mentioned method – Outcome: sha2 uses around 2-3% more memory than sha1 (metadata caches) – We clearly know the cost of this feature 26June 29th 2015

Benchmarking: comparison integration This feature is also integrated in Jenkins – CvmfsBranchComparison project CvmfsBranchComparison – It provides a user-friendly interface to specify all necessary parameters 27June 29th 2015

Automatic Integration: Jenkins So far we have already mentioned important automatic integration projects on Jenkins: – All our benchmarks run once a day – Branch comparison through benchmarking Triggered on demand – Test coverage checking once a week 28June 29th 2015

Automatic Integration: projects 29June 29th 2015

Automatic Integration: documentation Enable the “make doc” functionality to generate Doxygen documentation in HTML formatDoxygen – After the generation the files are uploaded to a web server It is also fully integrated on Jenkins You can find the latest documentation herehere 30June 29th 2015

Automatic Integration: documentation 31June 29th 2015

Automatic Integration: GHPRB GitHub Pull Request Builder (GHPRB) is a Jenkins plug-in that automatically builds a recently pushed pull request and runs the unit tests on it GitHub Pull Request Builder – Previously presentedpresented We spotted a security leak – It did not check GitHub’s secret to be sure that the message was not sent by someone else 32June 29th 2015

Automatic Integration: GHPRB I wrote a pull request fixing the security leak and has recently being approvedpull request Which means we can now safely use that plug-in In order to make it work we would need to change the Jenkins access policy 33June 29th 2015

Automatic Integration: arm64 platform Recently we have successfully compiled CernVM-FS for arm64 – We could build it in a shared arm64 slave – We would need a dedicated arm64 machine for testing and debugging in order to fully support it It could be potentially invasive 34June 29th 2015

Lessons learnt so far How to effectively use all these tools Testing allowed us to find a few bugs in the code Testing methodology – Usage of mocking services for a clearer and easier testing development Benchmarking is important because: It allows us to quantify the cost of changes Spot long-term degradation of performance How to work in open source projects – Specially using distributed version control systems The automatic integration is important 35June 29th 2015

36June 29th 2015

Appendix: Jenkins wish list Cobertura plug-in – Publish test coverage results GHPRB plug-in – Automatically build and test a pull request Remove CERN log-in screen to receive POSTs notifications from GitHub 37June 29th 2015