1. Page 1 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Chapter 8 Dispelling the Myth of Component Evaluation

2. Page 2 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Overview qIntroduction qMulti-criteria evaluation qExploding the myth of component evaluation qMulti-attribute evaluation and beyond

3. Page 3 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Introduction q Components: l Independently deployed software implementations. q Assemblies: l Aggregations of components that provide integrated behavior. q Selection: l Choosing one component over another involves selection. q Evaluation: l Formalized process of quantifying human judgment by assigning value to choices.

4. Page 4 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Evaluation q If… l The quality of software components determines of the quality of the composed system, q Then… l CBSE must provide techniques to reliably and repeatedly select high quality components. q And it follows from this that … l Component evaluation is a distinguished CBSE activity, with distinguished workflows and techniques.

5. Page 5 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Evaluation Attributes select vendorcomponent healthfunctionusabilityreputation composed attribute basic attribute qualitative dependency

6. Page 6 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Evaluation Attributes select vendorcomponent healthusabilityfunctionalityreputation judgmentusability indexjudgment#menu items non-repeatable judgment essential judgment

7. Page 7 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Genus: Preference Structure-based Evaluation q A preference structure. l This is the model of the decision. l A preference structure emerges when we express preference relations in terms of attributes. q An aggregation technique. l This is the tool that generates interpretations of the model.

8. Page 8 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Preference Relation q P(x, y), strict preference: l States that x is strictly preferred to y. q I(x, y), indifference: l States that neither x nor y is preferred. q R(x, y), incomparability: l States that x and y are incomparable. q For example, we might define a preference relation: S(x, y) = P(x, y) È I(x, y)

9. Page 9 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Species: Multi-attribute Utility Evaluation q The species can be seen through its formulaic expression, in which each evaluation attribute g k Î G is defined as the triple: U x = S w k · u k (g k (x)) q U x denotes the overall utility of component x, q u k denotes a transform function that maps the scale of attribute measure g k to a universal utility scale u k qw k denotes the substitution rate for g k

10. Page 10 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Multi-attribute Utility q The preference structure most frequently associated with multi-attribute utility is: S(x, y, g) = P(x, y, g) È I(x, y, g) P(x, y, g) « U x > U y I(x, y, g) « U x = U y Which states that x is preferred to y if it has a higher utility, and x and y are indifferent if they have the same utility.

11. Page 11 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Simple Utility Transform Functions Usually usability index utility 100 0 u(usability) # menu items utility 100 0 u(functionality)

12. Page 12 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Exploding the Myth of Component Evaluation q An Assembly: l Reflects the convenience in representing the composition of commercial components as systems, subsystems, sub-subsystems, and so forth. l The scope of a system, or its relative position in a hierarchy of systems is not material to what follows, we will use the term ‘assembly’ in place of ‘system’. q That is, commercial components are assembled into assemblies.

13. Page 13 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Assemblies and Components q Assemblies, once they exist, and commercial components will exhibit a variety of properties: l functionality, l reliability, l usability, l and so forth. qThe properties of an assembly are determined, in some way, by the properties of the components themselves

14. Page 14 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Assembly Properties C1C1 C2C2 P1P1 P2P2 A PAPA P A = D (P 1, P 2 ) Assembly Properties Determined by Component Properties

15. Page 15 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Satisfaction of Normative Abstract Interface C1C1 P1P1 P E1 A PAPA P A = D (P E1, P E2 ) satisfies

16. Page 16 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Accommodating Variance C1C1 P1P1 P E1 A PAPA P A = D (P E1  P 1, P E2  P 2 ) P 1  P E1 sat

17. Page 17 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation The Inevitability of Hidden Properties C1C1 P1P1 P E1 A PAPA P A = D (P H1, P H2, P E1  P 1, P E2  P 2 ) P 1  P E1 sat P H1 P E1  P H1 = 

18. Page 18 Building Reliable Component-based Systems Chapter 8 - Dispelling the Myth of Component Evaluation Fragment of Design Search Space