Barmak Meftah VP, Engineering Fortify Software Software Security at the Source
Why Benchmarks? A good and hopefully objective gauge for comparing bug detection tools or for that matter any piece of infrastructure software Should have 3 attributes: Measure what end-users care about (results that can be easily consumed, performance, and accuracy of output) Scenarios that users care about and is close to real life situations (TPC benchmarks are a great example) Start with known bugs (maybe open source projects)