1. Software Architecture Prof.Dr.ir. F. Gielen
Architecture Trade-off Analysis Method
ATAMSM
Vakgroep Informatietechnologie – IBCN

2. Context for Evaluating Architectures.
Before we consider specific methods for evaluating software architectures we will consider some context :
SA will tell you important properties of a system even before it exists
Architects know the effects (or can estimate) of design (architectural) decisions
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

3. Why Evaluate Architectures?
All design involves trade-off’s.
A software architecture is the earliest life-cycle artifact that embodies significant design decisions and trade-offs.
The earlier that risks are identified, the earlier that mitigation strategies can be developed potentially avoid the risks altogether.
The earlier that defects are found, the less it costs to remove them.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

4. Cost / Benefits of Evaluation of SA
Staff time that is required of the participants
AT&T (Lucent) performed ~300 full scale SA Evaluations
Average cost 70 staff-days
ATAM method averages 36 staff-days + stakeholders
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

5. Cost / Benefits of Evaluation of SA
Financial Benefits
AT&T estimated that the 300 Evaluations saved 10% in project costs
Anecdotal information of evaluations savings
Company avoided multi-million dollar purchase when evaluation showed inadequacy
Anecdotes on evaluations that should have been done
Estimate of rewrite of system would take two years; it took three rewrites and seven years
Large engineering DB system; design decisions prevented integration testing. Cancelled after $20,000,000 invested
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

6. Dimensions of complexity
Higher technical complexity
- Embedded, real-time, distributed, fault-tolerant
- Custom, unprecedented, architecture reengineering
- High performance
An average software project:
people
month duration
- 3-5 external interfaces
- Some unknowns & risks
Defense Weapon System
Telecom
Switch
National Air Traffic
Control System
Commercial
Compiler
Embedded
Automotive
Software
Large-Scale
Organization/Entity
Simulation
Lower
management
complexity
- Small scale
- Informal
- Single stakeholder
- “Products”
CASE Tool
Higher
management
complexity
- Large scale
- Contractual
- Many stake holders
- “Projects”
Small Scientific
Simulation
IS Application
Distributed Objects
(Order Entry)
Enterprise IS
(Family of IS
Applications)
Defense
MIS System
IS Application
GUI/RDB
(Order Entry)
Business
Spreadsheet
Lower technical complexity
- Mostly 4GL, or component-based
- Application reengineering
- Interactive performance
Walker Royce, Rational
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

7. Non-Financial Benefits
Forced Preparation for the review:
Presenters will focus on clarity
Captured Rationale
Evaluation will focus on questions
Answering yields “explanations of design decisions”
Useful throughout the software life cycle
After the fact capturing rationale much more difficult
“Why was that done?”
Validation of requirements
Discussion of how well a requirement is met opens discussion
Some requirements easy to demand, hard to satisfy
Improved architectures
Iterative improvement
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

8. Planned or Unplanned Evaluations
Normal part of software life cycle
Built into the project’s work plans, budget and schedule
Scheduled right after completion of SA (or ???)
Planned evaluations are “proactive” and “team-building”
Unplanned
As a result of some serious problem
“mid-course correction”
Challenge to technical authority of Team
Unplanned evaluations are “reactive” and “tension-filled”
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

9. Properties of Successful Evaluations
Clearly articulated goals and requirements
Controlled Scope
Cost-effectiveness
Key personnel availability
Competent Evaluation Team
Managed Expectations
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

10. Outputs of an Architecture Evaluation
Prioritized statement of quality attribute requirements
Mapping of approaches to quality attributes
Risks and non-risks
Risks are potentially problematic architectural decisions
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

11. Documenting risks and non-risks consists of:
An architectural decision (that has not been made).
A specific quality attribute response being addressed by that decision.
A rationale for the positive or negative effect that the decision has on satisfying the quality attribute.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

12. Technological Risk Levels
Amount of uncertainty
1. DISASTER
Believed to have never been done by anyone, anywhere in the world!
2. VERY MAJOR
Done by somebody, somewhere, once
3. MAJOR
Done elsewhere a few times, but never
yet in-house
4. MEDIUM
Done commonly or by people who now
are in-house
5. MINOR
Done often, including in-house
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

13. QA response and consequence:
Example of a Risk
Decision to be made:
The rules for writing business logic modules in the second tier of your three tier client-server system are not clearly articulated.
QA response and consequence:
This could result in replication of functionality thereby compromising modifiability of the third tier.
Rationale for the negative effect:
Unarticulated rules for writing business logic can result in unintended and undesired coupling of components
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

14. Architecture Tradeoff Analysis Method (ATAM)
“We evaluate the services that anyone renders to us according to the value he puts on them, not according to the value they have for us.” Friedrich Nietzche
Evaluating an architecture is a complicated undertaking.
Large system  large architecture
ABC + Nietzche’s quote yields:
A software system is intended to support business goals and
Evaluation needs to make connections between goals and design decisions
Multiple stakeholders  acquiring all perspectives requires careful management
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

15. Architecture Tradeoff Analysis Method
ATAM is an architecture evaluation method that:
focuses on multiple quality attributes
illuminates points in the architecture where quality attribute trade-offs occur.
generates a context for ongoing quantitative analysis.
utilises an architecture’s vested stakeholders as authorities on the quality attribute goals.
The purpose of ATAM is:
to assess the consequences of architectural decisions in light of quality attribute requirements and business goals.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

16. Figure taken from http://www.sei.cmu.edu/ata/ata_method.html
Conceptual Flow ATAM
Figure taken from
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

17. Participants in ATAM Evaluation Team Project Decision makers
group external to the project
3-5 persons
need to be recognized as unbiased outsiders
Project Decision makers
the architect, customer, project manager
people empowered to speak for the development project
with the authority to make decisions
Architecture stakeholders
including developers, testers, field engineers…,
users, customers
builders of systems interacting with this one.
Find at least 10 of them !
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

18. Evaluation Team Roles and Responsibilities
Team leader
Sets up evaluation (set evaluation contract)
forms team
interfaces with client
Oversees the writing of the evaluation report
Evaluation leader
Runs the evaluation
Facilitates elicitation of scenarios
Administers selection and prioritization of scenarios process
Facilitates evaluation of scenarios against architecture
Scenario scribe
Records scenarios on a flip chart
Captures (and insists on) agreed wording
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

19. Evaluation Team Roles and Responsibilities
Proceedings scribe
Captures proceedings in electronic form
Raw scenarios with motivating issues
Resolution of each scenario when applied to the architecture
Generates a list of adopted scenaios
Timekeeper
Helps evaluation leader stay on schedule
Controls the time devoted to each scenario
Questioner
raises issues of architectural interest stakeholders may not have thought of
Process observer
Keeps notes on how the evaluation process could be improved
Process enforcer
helps the evaluation leader stay “on process”
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

20. A concise presentation of the architecture
Outputs of the ATAM (1/2)
A concise presentation of the architecture
Frequently there is too much information.
ATAM will force a “one hour” presentation of the architecture forcing it to be concise and clear.
Articulation of Business Goals
Quality requirements in terms of collections of scenarios
Mapping of architectural decisions to quality attribute requirements
A set sensitivity and tradeoff points
Eg. A backup database positively affects reliability (sensitivity point with respect to reliability)
However it negatively affects performance (tradeoff)
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

21. A set of risks and non-risks
Outputs of the ATAM (2/2)
A set of risks and non-risks
A risk is an architectural decision that may lead to undesirable consequences with respect to a stated quality attribute requirement
A non-risk is an architectural decision that, after analysis, is deemed to be safe
Identified risks can form the basis of a “architectural risk mitigation” plan
A set of risk themes
Examine the collection of risks produced looking for themes that are the result of systematic weaknesses in the architecture
Other outputs – secondary
More documentation, rationale, sense of community between stakeholders, architect, …
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

22. ATAM Phases
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

23. Team leadership/ key project decision makers Over a few weeks
ATAM planning
Phase
Activity
Participants
Duration
Preparation
Team leadership/ key project decision makers
Over a few weeks 1
Evaluation
Evaluation team and project decision makers
1 day + hiatus of 2 to 3 weeks 2
Evaluation with Stakeholders
Ditto + stakeholders
2 days 3
Follow-up
Evaluation team and client
1 week
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

24. Issues that must be resolved in Phase 0:
Phase 0: Partnership
Client : someone who can exercise control of the project whose architecture is being evaluated
Perhaps a manger
Perhaps someone in an organization considering a purchase
Issues that must be resolved in Phase 0:
The client must understand the evaluation method and process. (train the customer)
The client should describe the system and architecture. A “go/no-go” decision is made here by the evaluation team leader.
Statement of work is negotiated.
Issues of proprietary information resolved.
IPR & NDA
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

25. Forming the evaluation team Holding an evaluation kickoff meeting
Phase 0: Preparation
Forming the evaluation team
Holding an evaluation kickoff meeting
Assignment of roles
Roles not necessarily one-to-one
The minimum size evaluation team is 4 (3+1)
One person can be process observer, timekeeper and questioner
Team leader’s responsibilities are mostly “outside” the evaluation. He can double up. (Often the evaluation leader.)
Questioners should be chosen to represent the spectrum of expertise in performance, modifiability, …
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

26. Phase 1: Activities in Addition to the steps
Organizational meeting of evaluation team and key project personnel
Form schedule
The right people attend the meetings
They are prepared (know what is expected of them)
They have the right attitude
Besides carrying out the steps the team needs to gather as much information as possible to determine
Whether the remainder of the evaluation is feasible
Whether more architectural documentation is required
Which stakeholders should be present for phase 2
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

27. ATAM steps: phase 1
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

28. Evaluation leader presents the ATAM to all participants
Step 1: Present the ATAM
Evaluation leader presents the ATAM to all participants
To explain the process
To answer questions
To set the context and expectations for the process
Using standard presentation discuss ATAM steps in brief and outputs of the evaluation
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

29. Step 2: Present Business Drivers
Everyone needs to understand the primary business drivers motivating/guiding the development of the system:
NABC
Need:
What is the important customer and market need?
Approach:
What is your unique approach for addressing this need ?
Benefits:
What are the benefits (per cost) from this approach ?
Competition:
How are those benefits superior to the competition and the alternatives ?
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

30. Step 2: Example – Video on demand
Need:
Movie rental is a 500 Mio Euro business. The part people dislike is to return tapes and late fees.
Approach:
We will provide VOD via the cable system with access to all the tittles of MovieMAX. The system uses existing channels and hardware. Customers need no new investments and pay the same price for a movie as in the rental shop.
Benefits:
End-user: no need to return movies; no more late fees. Same functions as with a DVD player: fast forward, trailers, ..etc.
Customer: Higher revenue per movie with higher margin; 20% market share expected.
Competition:
competition : we have patented the distribution and VCR like features for VOD
alternatives : on-line rentals have higher handling costs (0.75 Euro per movie). Sending the tape back is as inconvenient as returning it.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

31. Step 2: Present Business Drivers in practice
A project decision maker presents a system overview from the business perspective
The NABC.
The system’s core functionality
Relevant constraints:
technical, managerial, economic, or political
The architectural drivers
the quality attributes that shape the architecture
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

32. Step 3: Present the Architecture (1/2)
The lead architect makes the architecture presentation at the appropriate level of detail (~20 slides; 60 minutes)
Architectural drivers and existing standards/models/approaches for meeting these(2-3 slides)
Important Architectural Information (4-8 slides)
Context diagram
Module or layer view
Component and connector view
Deployment view
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

33. Step 3: Present the Architecture (cont)
Architectural approaches, patterns, or tactics employed (3-6 slides)
Use of COTS (commercial off the shelf) products
Trace of 1-3 most important use cases scenarios
Trace of 1-3 most important change scenarios
Architectural issues/risks with respect to meeting the driving architectural requirements
Glossary
Should cover technical constraints such as operating system, hardware and middleware
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

34. Step 4: Identify Architectural Approaches
The ATAM analyzes an architecture by focusing on its architectural approaches.
Styles and patterns are captured here by the evaluation team.
The evaluation team should explicitly asked the architect to name the patterns, styles and approaches.
The list is cataloged and documented.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

35. Step 5: Generate Quality Attribute Utility Tree
Quality attributes are articulated in detail:
Evaluation team works with project decision makers to elicit the most important quality attributes as scenarios.
Root: Utility is an expression of the benefit to the user or customer.
Level 1: Quality attributes are the second level. The initial set comes from the business drivers (Step 2).
Level 2: Attribute refinements.
Level 3: Short scenarios (stimulus, context, response)
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

36. Quality Attribute Utility Tree Example
Figure taken from Linda Northrop’s presentation for CSEE&T 2004, March 2004
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

37. Step 5: Generate Quality Attribute Utility Tree
Scenario prioritisation
Project decision makers give a relative ranking based on importance (business goal).
Architect gives a ranking based on difficulty to satisfy the scenario. (technical goal)
Each scenario has a (H,H), (H,M),…(L,M),(L,L) priority assigned.
Based on this a selection of scenarios is made for the purpose of evaluation.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

38. Step 6:Analyze Architectural Approaches
The evaluation team examines the highest priority scenarios one at a time and the architect is asked how the architecture supports each one.
architectural decisions
sensitivity points
tradeoff
risks,
Key is to have sufficient information to link architectural decisions to quality requirements
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

39. Example of Scenario Analysis :JBoss
Scenario U1: Deploy a modified library to the JBoss application server within five minutes.
The Results of Analysis of Scenario U1
D1: Using the unified class loaders to facilitate hot redeployment.
D2: Using the deployment ScannerThread to periodically check for
changes in the deployment directory.
D3: The deployment unit is not locked during the deployment.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

40. An Example of Scenario Analysis
T1: Tradeoff between reusability and the support of multiple versions of
components. The unified class loaders support the reuse of the
components among different deployment packages at run-time, but
they limit the support for the multiple versions of the components .
S1: If the interval for checking is too long, the total deployment time will
be longer. If the interval for checking is too short, CPU resources
will be wasted.
R1: If the copy takes too long time to complete (large library or remote
copy), JBoss may deploy an incomplete library.
R2: If the library is changed during the deployment, JBoss may deploy a
library in an inconsistent state.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

41. ATAM steps: phase 2 with all stakeholders
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

42. Step 7: Brainstorm and Prioritize Scenarios
The Hiatus
Evaluation team distills what has been learned and informally contacts architecture for more information where needed
Scenario brainstorming
Take input from a large group
Compare this list with the utility tree of phase 1
prioritize the stakeholder scenario list
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

43. Step 8: Analyze Architectural Approaches
Similar to step 6
Applied to the highest ranking scenarios of step 7
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

44. Step 9: Present results Present the outputs Identify risk themes
focus on utility, risks, sensitivity and tradeoff
Identify risk themes
group risks into categories
identify impacted business drivers
define mitigation plan
action before the risk becomes a problem
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

45. Figure taken from http://www.sei.cmu.edu/ata/ata_method.html
Summary: ATAM Flow
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN
Figure taken from

46. Summary: When to Use the ATAM ?
The ATAM can be used throughout the life cycle when there is a software architecture to evaluate.
ATAM can be used:
after an architecture has been specified but there is little or no code
to evaluate architectural alternatives
to evaluate the architecture of an existing system
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

47. The result is improved architectures.
ATAM benefits
The benefits of performing ATAM evaluations include:
clarified quality attribute requirements
improved architecture documentation
basis for architectural decisions
identification of risks early in the life cycle
increased communication among stakeholders
The result is improved architectures.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN