1 Example application: source code analysis 125 file types; 8029 files; 4689 non-Java; 1112 svn revisions

2 build scripts version history spreadsheets databases config files web pages bug reports software repository parsers query engine analyst dash board IDE plugin excel add-in source code developer manager Querying Software Artefacts

3 The problem design query language and engine for accessing vast repository of different types of source artefact libraries of queries: tailor framework to different types of artefact

4 Tough problem! Difficulties: - does not scale - efficient queries extremely hard to write - specific to one kind of source artefact Dozens of attempts, in industry and academia since 1984: databases, prolog, domain-specific query languages 18 man-years of research at University of Oxford to discover ingredients of solution 15 man-years to implement an industrial product 3 patents pending, several more in pipeline

5 SemmleCode: the power of.QL

6 The query language.QL  Object-oriented, for creating libraries of queries  Recursive queries, as in logic programming  Familiar syntax to Java and SQL developers  On top of any traditional relational database  Syntax-highlighting, error-checking and auto-completion

7 How it works.QL library.QL query RDBMS procedural SQL java / jar bytecode for search XML files template for RDBMS Semmle optimiser

8 Demo  The source we shall explore:  Alfresco: Enterprise Content Management  Spring: Java/JEE Application Framework  Builds on Tomcat, JBoss, … Demo parts: out-of-the-box writing your own queries querying XML config files Vital statistics: Java methods 6647 Java types 516 XML files

9 Using SemmleCode out-of-the-box 115 pre-packaged queries Find common bug patterns: e.g. compareTo/equals, cloning, serialisation, internationalization Compute metrics: 42 different metrics, including Robert Martin’s package metrics Examine dependencies: e.g. cyclic package dependencies Visualization: pie charts, bar charts, tables, graphs, warnings/errors - easy navigation to source - exportable for generating reports

10 Writing queries of your own: select from Method m where m.fromSource() and m.hasName("compareTo") and not m.getDeclaringType(). getAMethod().hasName("equals") select m, "missing equals?" In general: from where select

11 Writing queries of your own: aggregates select sum (CompilationUnit cu | cu.fromSource() | cu.getNumberOfLinesOfCode()) In general: agg( T 1 x 1, …, T n x n | condition | expr )

12 Writing queries of your own: recursion from RefType s, RefType t, RefType it where it.hasName("PasswordInputTag") and it.hasSupertype*(s) and it.hasSupertype*(t) and t.hasSupertype(s) select t,s In general, can write recursive predicate definitions

13 Queries in.QL from-where-select autocompletion, typechecking, emptiness tests aggregates arbitrary nesting, no group-by needed recursion implicit with chaining; or explicit

14 Defining new classes in.QL class ClassAttribute extends XMLAttribute { ClassAttribute() { this.getName()="class" } string getClassName() { this.getValue() = result } RefType getType() { result.getQualifiedName() = this.getClassName() } predicate noType() { not exists(this.getType()) } } from ClassAttribute ca where ca.noType() and ca.getClassName().matches("org.alfresco%") select ca, ca.getClassName() + " not found"

15 Classes in.QL classes are logical properties “constructor” specifies characteristic property methods body is relation between this, result and parameters more than one result allowed predicates methods without a result body is relation between this and parameters

16 The key points of.QL classes are predicates inheritance is implication nondeterministic expressions recursion with super-simple semantics syntax familiar to SQL and Java programmers designed for creating libraries of queries excellent error checking and IDE integration

Concluding remarks

18 Couldn’t you use LINQ instead of.QL?  Different design goals: ORM versus libraries of queries  LINQ does not provide recursion  LINQ cannot do the optimisations across multiple queries that are key to efficiency in.QL “Fortunately, there is light in the darkness. Based on decades of programming language research, the brilliant team at Semmle has created an elegant, industrial strength object-oriented query language called.QL with full support for recursive queries and aggregation….QL has all the requisites to become a runaway success.” (Erik Meijer, Creator of LINQ, Microsoft)

19 Too good to be true? Jeff Ullman, 1991: It is not possible for a query language to be seriously logical and seriously object-oriented at the same time. key breakthroughs are Semmle’s proprietary technology: - design of.QL - optimisations on “bytecode for search”

20 Wrapping up Java is not enough source code analysis tools must process a multitude of artefacts libraries of queries a means to achieve such heterogeneous tools.QL object-oriented queries over trees and graphs made fast and easy