An Evaluation of BLAST John Gallagher CS4117

Overview BLAST incorporates new, fascinating and complex technology. The engine and external components are evolving monthly. BLAST can be used successfully on carefully scoped (systems) programs. BLAST exists to serve software model checking research and make it tangible. It is not a good developer tool.

Criteria How easy is it to generate the input? How easy is it to generate the output? How accurate is the output? How precise is the output? How broad are the applications of the tool? What are the alternatives? How much time or process does the tool add or remove from software development? Opportunity Cost

The Good – BLAST Framework BLAST internals illustrate interesting working examples of compiler technology: int main() { int i, x, y, ctr; x = ctr; ctr = ctr + 1; y = ctr; if (x == i) { assert (y == i + 1); } } :-)

The Good – BLAST Framework Based on very recent research (earliest work cited in corresponding paper was from 2002). Given good input, correct use of BLAST’s myriad of runtime options, and good specification, BLAST performs well. Able to find real bugs that were fixed based on counterexample information that no other tools were providing.

The Good – BLAST Framework Running time figures and number of real counterexample traces make BLAST look good. They are the result of a solid framework. Favorable position of having little competition in terms of results achievable. ProgramLines pre- processed Time (mins) Predicates Total Average kbfiltr 12k floppy 17k diskprf 14k cdaudio 18k parport 61k parclss 138k

Future – BLAST Framework Using interpolation in predicate discovery was a step forward. This reduced the number of abstract predicates to track. Lazy Interpolation-Based model checking integrates interpolants further into the engine. It is being explored by BLAST Kenneth McMillan and BLASTer Ranjit Jhala. Expression abstraction may yet be able to represent more expressive C patterns. Basic user-mode program specifications cannot be verified.

The Bad – BLAST Usability Not easy to install (Specific path dependencies, purported Cygwin compatibility a fallacy) Not easy run BLAST Query Language pre-processor yields bad output, contains no usage information BLAST Query Language pre-processor yields bad output, contains no usage information BLAST main executable has 232 lines of usage information, some of which is completely broken, some of which has significant impact on results. BLAST main executable has 232 lines of usage information, some of which is completely broken, some of which has significant impact on results.

The Bad – BLAST Usability Output with debugging suppressed contains about.5% useful information. With debugging added, that number is significantly less. (Even a developer would be hard pressed to decipher over one-hundred consecutive lines of “: is the zero DD”). The tool’s specification weaver does not correctly remember source code lines, so useful output requires extensive corresponding grep into the source.

BLAST – Recommendations Integrate BLAST into development process early. Take advantage of BLAST’s typically fast run times to perform verification as a post- compile operation. Take note when BLAST output changes dramatically, and find the corresponding change in source.

BLAST – Recommendations BQL (BLAST Query Language) is a good way to separate specification from implementation and eliminate BLAST checks that could turn into useless runtime costs Build tools around BQL specification weaver or improve it (ahh, the gift and the curse of open source).

Questions?

References The information shown here is based by corresponding paper on the BLAST tool, “Software Verification with BLAST”. A good list of references can be found there.