A few Billion Lines of code Later using Static Analysis to find Bugs in the Real World BY Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-gros, Asya Kamsky, Scott McPeak and Dawson Engle

Introduction Coverity is a brand of software development products, consisting primarily of static code analysis and dynamic code analysis tools. How static code analysis was commercialized ? What had to be changed for real world use ? How to adapt to the constant change in requirements ? What was the lesson learnt ?

Lab vs Industry The software was able to find bugs in a large code base easily with few simple analyses and tricks in a lab environment. In the real world, hundreds of programmers use the tool to check hundreds of different code bases. The types of errors, number of false positives, type of build are all different from what is predicted by lab results. The programmers do not know how the tool works, unlike testers in the lab who have a knowledge of the tools’ internal process.

Customer Interaction Two scenarios of customer interaction Initial trial Long term use The trial is a pre-sale demonstration that attempts to show that the tool works well on a potential customer’s code. Sales Engineers educate the customers about the tool. Usually happens over a period of 1 day or 2-3 days if code base is very large

Laws of Bug Finding You can’t check code you don’t see Ability to extract code from build No access to modify build Test Machine You can’t check code you can’t parse No standard compilers

BUGS Myth: More analysis is always good Do bugs matter? No, your tool is broken, that is not a bug Misunderstood errors/bugs are considered as false positives How to handle cluelessness ? Do not change the results after an upgrade Myth: More analysis is always good

QUESTIONS What are the lessons learnt ? How would you commercialize a bug finding tool ? Any experience using a bug finding tool ?

Thank You