1. BX Benchmarks Working Group Results BIRS Center, Dec. 3, 2013

2. Addressed Questions Which categories of bx benchmarks do exist? What´s the difference between a bx example and a bx benchmark? What are bx benchmarks expected to measure? What about formats for bx benchmark descriptions? [ Discussion of meaningfulness of „incrementality“ in state-based scenario ]

3. Categories of bx Benchmarks Functional bx benchmarks address selected features of bx approaches: – Properties of consistency relations – Types of transformation scenarios – Laws for bx approaches – [ Appropriateness for categories of data ? ] – [ Platform dependency ? ] Non-functional bx benchmarks: …

4. Categories of bx Benchmarks … Non-functional bx benchmarks address bx transformation characteristics like: – Runtime / memory consumption – Scaleability w.r.t. to size of data, spec. … – Size of propagated deltas – [ Loss of information after update propagation ] – …

5. bx Benchmark Properties A bx benchmark is a bx example (with an optional additional characterization w.r.t. to a bx taxonomy) that has a precise and executable definition of a binary consistency relation on source and target data (that can be used as an oracle for a bx application) an explicit definition of or a generator for input data elements (for a bx application) set of precisely defined update scenarios for certain input data elements set of executable metrics definitions (with an SE interpretation of the word „metrics“)

6. Further bx Benchmark Properties A „useful“ bx benchmark should capture the essence of certain aspects of a class of real- world bx application scenarios bx descriptions may exist on different levels of abstraction with varying degrees of dependence on a specific execution platform (PIM bx benchmark description, … )