1. PerOpteryx
Automatically Improve Software Architecture Models for Performance, Reliability, and Costs using Evolutionary Algorithms
Anne Martens Karlsruhe Institute of Technology (KIT), Germany Heiko Koziolek, Steffen Becker, Ralf Reussner
WOSP / SIPEW 2010
In the next twenty minutes, I will present the PerOpteryx approach to automatically improve software architecture models for performance and other qualities to you. PerOpteryx stands for Performance optimisation here, but is also the name of a bat. The idea is that the PerOpteryx approach explores the design space of software architectures just as a bat explores physical space with its sonar.
But before going into details, let us first set the context.
Prof. Dr. Max Mustermann | Name of Faculty

2. Software Performance Engineering
SDQ
Software Performance Engineering
3 sec
Transform
Solve C1 A C2 A
Change component and deployment
2.5 sec
Transform
Solve
There are many approaches around to predict performance properties based on software architecture models of a system. (click) We can automatically transform the software architecture models into formal performance model, such as a queuing network here, and evaluate these. Here, we predict a response time of 3 seconds.
Because we use the architecture model as a starting point here, we can also easily evaluate the effects of design decisions. For example, we can change the architecture by changing the component instantiation on the left and by adding a second server.
Our transformation chain (click) can tell us that this solution is better. The nice thing if we use these architecture models is, that we can devise an automatism (click) that varies the model for us within certain defined boundaries until it finds the design with the best performance.
However, are these results really useful?
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

3.  Optimise multiple criteria at once
SDQ
Not only Performance!
3 sec
p(fail) 0.01% $ 5700
Transform
Solve C1 A
Change component and deployment
2.5 sec
p(fail) 0.02%
$ 12000
Transform
Solve C2
However, are these results really useful? There are other quality attributes of software architectures, not just performance. Here, we have added probability of failure on demand and cost, for example measured as total cost of ownership. In this case, (click) the probability of a failure for a request to the system has increased, maybe because component C2 is more unreliable. Also, the overall cost has increased due to the second server.
This is not just bad luck, but we usually have a trade off between performance and other qualities.
We observe that we cannot handle performance separately (click), but we have to consider multiple criteria at once. A
 Optimise multiple criteria at once
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

4. Multicriteria Optimisation
SDQ
Multicriteria Optimisation
Architectural Candidate A
Time 5s
How can we optimise for multiple criteria at once? First, we can evaluate each architectural candidate for the quality criteria of interest (click). Here, we have architectural candidate A, for example the initial one of the previous slide with both components being deployed on one server (click). It has a cost of $ and a response time of 5 seconds.
$40K
Costs
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

5. Multicriteria Optimisation
SDQ
Multicriteria Optimisation
Generated & Evaluated A
Time 5s
If we specify how the architecture can be changed, we can generate and evaluate a number of candidates for their quality properties.
$40K
Costs
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

6. Multicriteria Optimisation
SDQ
Multicriteria Optimisation A
Time 5s B 3s C 2s
For a candidate (click) that we name B here, we observe (click) a response time of 3 seconds and cost of 20000$. For this other candidate C (click), we get (click) a lower response time of 2 seconds, but higher cost of $.
And what we see here is a common result: If we look at all these candidates, we do not have a single best candidate that has lowest response time and lowest cost. We observe the trade-off between performance and cost between B and C.
However, we can distinguish candidates that are suboptimal. Here, candidate A is worse than B and C in all criteria. It is thus called Pareto-dominated.
The candidates on this line (click) are called Pareto-optimal because they are not dominated by other solutions. These candidates are the ones with optimal trade-off.
The goal of multicriteria optimisation is to find this set of Pareto-optimal candidates. Based on these, the software architect can make well-informed trade-off decisions. Note that we can also take into account more dimensions. We consider three dimensions later, and it is unlimited in general.
Pareto-optimal
$20K
$33K
$40K
Costs
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

7. Related Work: Quality Optimisation
SEQ
Related Work: Quality Optimisation
Rule-based approaches: Single quality only
Parsons2008, Cortellessa2009, PerformanceBooster (Xu&Woodside2008), ArchE (McGregor2007)
Multicriteria evaluation: No improvement
Bondarev2007, Grunske2007
Optimisation: Limited degrees of freedom
ArcheOpteryx (Aleti2009), Canfora2005, Kavimandan2009, Sassy (Menascé2010)
So before we look at our approach in detail, let us look at related work.
There are several groups of related approaches. The first group improves software models for a single quality attribute using rules. For example, bottleneck resources are detected and mitigated by adding more resources. However, they only address a single criterion in isolation.
(click) the second group of approaches allows to evaluate several qualities of a single software model. However, no automated improvement is suggested, that means, no new candidate solutions are generated.
(click) Finally, the most related group of approaches do multicriteria optimisation for software models. However, they only address a limited set of ways in which the architecture can be changed. For example, only service selection in an SOA is addressed by Canfora and others.
(click) What is missing is the multicriteria optimisation that allows to explore many variation types in software design models, that is deployment, component selection, and other configuration.
Cortellessa (Lesegruppe) : Antipattern detection with MDSD for performance feedback, Xu / Woodside: Rule-based, automatic improvement of LQNs, no mapping back to design model. McGregor ArchE Modifiability tactics Parsons: Detect anti patterns in J2EE apps
Bondarev: Simulate given candidates and show Pareto curve Grunske: Framework for multicriteria evaluation, but no improvement
Aleti: Evolutionary for redundancy of components, reliability and cost. Missing: degrees of freedom are also part of the architecture, they are not considered. Canfora: QoS aware service composition with genetic algorithms Kavidaman: Deployment, minimise number of servers with bin packing
Missing: Flexible multicriteria optimisation at the design level
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

8. Flexible degrees of freedom
PerOpteryx Approach
Flexible degrees of freedom
Multiple qualities
Multi-criteria optimization
And here is what distinguishes our PerOpteryx approach. First and most importantly, we have a flexible and extendable definition of degrees of freedom, that is of what in the architecture can be changed.
We support several quality attributes to support trade off decisions. Currently, performance, reliability and cost.
Finally, we apply multicriteria optimisation techniques to automatically improve the architecture within the defined degrees of freedom and for the named qualities.
I will go into details on these points in the following.
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty

9. Design decision that can still be made
SDQ
Degrees of Freedom
Design decision that can still be made C
Variation point
Which instance to use for component type C?
Range of options
C1, C2, or C3
So, first, what is a degree of freedom? Degrees of freedom are design decisions that can still be made. It is a variation point in the software architecture. Everything that can be modelled, changed, and that affects any quality attribute could be considered here. Usually, the change does not change the functionality.
We can characterise a design decision by naming a variation point of the system (here: which implementation to use for component type C) and a set of options we have (here, we can choose among components C1, C2, and C3). Thus, the component selection for component type C is a degree of freedom in our architecture with range C1 to C3.
Degree of freedom
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

10. Types of Degrees of Freedom in CBSE
SDQ
Types of Degrees of Freedom in CBSE
Software
Component selection
Middleware selection
Component replication
Software configuration
Deployment
Allocation
Processing Rate
Number of Servers
After having defined the notion of degree of freedom, we have identified possible degrees of freedoms of component-based software architectures. We looked for degrees that should be changed by automated approaches to improve performance and other qualities. (describe and click)
Selection: blackbox, only exchange as a whole, no internals touched.
Our point here is, that optimisation approaches for software architectures should not be specific for certain types of degrees of freedom, but should support a generic notion of degrees of freedom so that arbitrary ones can be added here and be explored.
Software configuration also software stack configuration, settings of JVM,
Software configuration
Thread pool size, priorities, ...
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

11. Instances of Degrees of Freedom
SDQ
Instances of Degrees of Freedom
Degree
Matching Rule
...
Allocation
Each component
Component selection for D
Processor speed
Each server
Component selection
Search alternatives
Component selection for C C
Allocation of D D
For one specific system at hand, there will be several instantiations of the degree types. With a matching rule for each type ofdegree, we can automatically find the instantiations in the system. For example, ...
Note that this set of degrees to be optimised are specific to this system.
Now, after we created this representation of the architecture and the possible changes, we would like to set up an automated search.
Processor speed of server 1
Allocation of C
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

12. Search Problem Degree Component selection C Allocation C
SDQ
Search Problem
Degree
Component selection C
Allocation C
Speed server 1
...
Choice C2
Server1
2 GHz
...
evaluate
Response in 2.5 s
P(failure) 0.02%
Cost $6000
transform
Now, after we created this representation of the architecture and the possible changes, we can set up an automated search.
The starting point is an initial model of the system, with initial values chosen arbitrarily for each degree of freedom. Then, (click) we automatically detect the instances of the degree of freedom types.
We created the following automation: To generate a candidate, we choose values fore each degree (click). For each degree of freedom type, we need a model transformation that applies the chosen option to the model. Any type of degree of freedom can be considered here if a generic transformation is available that applies a chosen value to the architecture! Then, (click) we can evaluate this candidate model and get a prediction.
The combination of all possible choices of all degrees here (click) is the space we have to search for good solutions. The lower part is the tool chain that allows automation. The actual search problem or optimisation problem is found on the upper layer here (click) We can evaluate a certain set of choices. Then, our goal is to find a variable assignment within the limits of the degrees of freedom that has optimal quality.
evaluate
initial model
candidate model
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

13. Search Implementation
We have implemented the proposed approach in the PerOpteryx tool. Currently, we are using genetic algorithms to search for optimal architectures. More specifically, we use the NSGA-II algorithm as implemented in the Opt4J optmisation framework.
NSGA-II
[Deb2002]
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty

14. Palladio Component Model
Quality evaluation
Palladio Component Model
[Becker2007]
PCM2Cost
[Martens2010] 
Cost
PCM2DTMC
[Brosch2009]
Reliability
PCM2LQN
[Koziolek2008]
Performance
Our approach is based on the Palladio component model to model software architectures. Thus, the PerOpteryx approach created Palladio instances during the search. We can then evaluate the quality attributes with the Palladio analysis tools. These are…
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty

15. Case Study with PerOpteryx (1/2)
Now let us have a look at a case study we created to show the feasibility of our approach. Consider this software architecture, a so called business reporting system. It accepts report requests from a user via a web interface. Requests are dispatched to four reporting engines. They query the database to generated the reports.
In addition to static structure and behaviour (not shown), we need annotations for the three qualities (click). (Show performance, reliability, cost).
The system here has four different types of reports, thus 4 different job classes with varying resource demands. Users arrive in a closed workload with 50 users and think time 5 sec. All this has been modelled in Palladio.
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty

16. Case Study with PerOpteryx (1/2)
Component allocation
Processing rates
Component selection
Response Time
In our case study, we consider three degrees of freedom: (show in architecture)
The PerOpteryx approach determined the following three dimenional Pareto-front of optimal candidates. We have interpolated this set of candidates here. Explain axes.
Overall, … (statistics)
For a better understanding, lets have a look at a two dimensional figure.
1235 candidates
58 Pareto optimal
8h running time
POFOD
Costs
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty

17. Case Study with PerOpteryx (2/2)
SDQ
Case Study with PerOpteryx (2/2)
RT: 1.34 s
POFOD: 5.2E-4
Cost: 69.83
Only four, but faster servers Different Webserver
RT: 2.2 s
POFOD: 6E-4
Cost: 98
Axes
Legende
(click) initial candidate here
(click) one optimal as example here.
With these results, the software architect now knows the optimal trade-off candidates: all blue ones are interesting. Now, a decision can be made for one candidate by weighting the different qualities.
Chosen optimal: Deploy on four servers, reporting and database components share a server, webserver and dispatcher together on one.
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

18. ? Performance heuristics Requirement support More degrees of freedom
Future Work
Performance heuristics
Requirement support
More degrees of freedom
Short term
Handle uncertainty of predictions
QoS process integration
Long term ?
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens

19. Flexible degrees of freedom
Conclusions
Automated Architecture Improvement
Flexible degrees of freedom
Multiple qualities
Multi-criteria Optimization
To conclude: (click) I have presented the automated software architecture improvement approach PerOpteryx to you. The main aspects of PerOpteryx are the following
(click) First, we have a flexible notion of degree of freedom of the architecture that defines the search space. Thus, our approach is not limited to certain degrees, such as allocation only, but covers several and can be extended to more.
(click) secondly, PerOpteryx supports the evaluation of several quality attributes to support trade off decisions. Currently, these are performance, reliability, and cost, and they can be evaluated with the Palladio tools.
(click) Finally, we apply multicriteria optimisation techniques to automatically improve the architecture within the defined degrees of freedom and for the named qualities.
(click) Implementation and case study available here
Motivation – Related Work – Approach – Case Study – Future Work –Conclusion
Anne Martens
Prof. Dr. Max Mustermann | Name of Faculty