1. Architecture Review Board
©USC-CSSE

2. Outline
USC CS577 ARB Concept
Participants
Procedures
Results

3. The Incremental Commitment Spiral Model (ICSM)
4 Key Principles:
Stakeholder value-based system definition and evolution
Incremental commitment and accountability
Concurrent system and software definition and development
Evidence and risk-based decision making
First of all, ICSM or the Incremental Commitment Spiral Model, which is a process model. One of the most important ingredient of the ICSM is the opportunity and risk that will determine course of action in the development.

4. ICSM for 24-week e-services projects

5. USC CS577 ARB Participants
Project Team
Everybody presents something
Reviewers
Clients
Instructors, TAs, and Graders
Each presentation takes 80 minutes.

6. (c)USC-CSSE

7. ARB/milestones for one-semester team
FCR/DCR ARB: Oct 15th, 17th, and 19th
Based on DC package
Focus on DCR success criteria
CCD: November 14th
Core Capability Drive-through
Client(s) will have hands-on experience on your core capabilities
TRR: November 26th, 28th, 30th
Based on AsBuilt package

8. (c)USC-CSSE

9. ARB/milestones for two-semester team
FCR ARB: October 15th, 17th, and 19th
Based on preliminary FC package
Focus on FCR success criteria
Hands-on client feedback session: November 14th
DCR ARB: November 26th, 28th, 30th
Based on draft DC package
Focus on DCR success criteria

10. ARB Review Success Criteria
FCR
For at least one architecture, a system built to arch. will:
Support the Ops Concept
Satisfy the Requirements
Be faithful to the Prototype(s)
Be buildable within the budgets and schedules in the Plan
Show viable business case
Most major risks identified and resolved or covered by risk management plan
Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR)
DCR
For the selected architecture, a system built to the arch. will:
Be buildable within the budgets and schedules in the Plan
All major risks resolved or covered by risk management plan
Key stakeholders committed to support full life cycle

11. Commitment Review Success Criteria
CCD
TRR / OCR
Determine whether client needs anything further to ensure successful Transition and Operation
Changes in priorities for remaining features?
Changes being made to operational procedures?
More materials needed for training?
Changes in system data or environment we should prepare for?
Anyone else who should experience CCD?
Show value
Product works as expected (or not with noted exceptions)
Product will help users do job
Show quality development
e.g. relevant parts of your
IOC documentation
Process
Show sustainability
e.g. support requirements/plans
Transition plan & status
Show confidence that product is/will be ready to be used
e.g. Transition plan & status
See also value

12. Team Preparation for ARB Reviews
Before ARB week
Within-team Dry run of presentations and demo
Further dry runs as necessary
During ARB week
ARB Presentation and discussion
Follow-up team discussions, client discussions
After ARB week
Monday: FC packages due (Two-semester team)
Monday: DC packages due (One-semester team)
Before ARB week
Within-team Dry run of presentations and demo
Further dry runs as necessary
ARB Week
ARB Presentation and discussion
Follow-up team discussions, client discussions
Week+1
Monday: FC packages due
Monday: DC packages due

13. Grading Criteria (70 points)
Quality of Presentation (10 points)
Quality of Project (40 points)
Progress (10 points)
Consistency and project synchronization (5 points)
Time management (5 points)

14. FCR ARB Session Outline Architected Agile Team
(x,y): (presentation time, total time)
(5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions;
(10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; desired capabilities and goals
(10,10) Prototype. Most significant capabilities [buying information](especially those with high risk if gotten wrong) Need to show additional progress addition to the first team prototype presentation
(5, 10) Requirements. Most significant requirements and its priorities level
(10, 10) Architecture. Top-level physical and logical architecture; Use case diagram, status of NDI/reuse choices
(5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation
(10, 10) Feasibility Evidence. Business case (beginnings, including benefits analysis); NDI analysis results (what are the COTS you are considering/considered); major risks;
(5, 5) QFP. Traceability Matrix and summary; Quality Management Strategy; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix, technical debt
(20) Things done right; issues to address (Instructional staff)
Do not forget your slide number
Each chart MUST have information specific to your project

15. DCR ARB Session Outline NDI/ NCS Team (1 semester)
(x,y): (presentation time, total time)
(5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations
(10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals
(5 , 5) WinWin Agreements. Agreed Win conditions in each category
(10,10) Prototype/ Product Demo. Most significant capabilities, NDI/NCS integration
(5, 5) Architecture. Top-level physical and logical architecture; (if applicable)
(10, 10) Life Cycle plan. Life cycle strategy; focus on Development phase & transition increment; key stakeholder responsibilities; project plan; resource estimation
(10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence; 3 personas
(5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix; Quality Management Strategy; Technical Debt
(20) Things done right; issues to address (Instructional staff)
Do not forget your slide number
Each chart MUST have information specific to your project

16. Specfics for DEN students Team’s strong points & weak points
List at least one item for each of the following
List your team’s strong points
Operational view
Technical view
List your team’s weak points
Identify specific technical concerns & possible solutions
Identify operational risks & possible mitigation
Sources of observations
Team activities, package evaluation, WinWin negotiation, and etc.

17. QFP – Defect Identification Review
Traceability Matrix
For each document section, UML model, and etc. identify the following
type of review you used (peer review, agile artifact review, and etc. )
Other form of defect identification, e.g., grading, client feedback, etc.
Current Defect Injection and Removal Matrix
Current, total defect information from your progress report
Briefly explain techniques, tools your team is using for quality management, configuration management.
Is it useful? Improvement?

18. ARB Session Outline DCR Similar format to FCR, different focus:
Less time for OCD, Prototype
More time for Architecture, Plans
Details TBD based on FCR ARB experience
General rule on focus: emphasize your project's high risk areas
At FCR (in most cases) this will involve establishing the operational concept (including system analysis)
At DCR (in most cases) this will involve the system design and development plan (especially schedule)

19. ARB Packages
bring at least 2 copies of your ARB presentation (2 or 4 slides per page)
Post your presentation and artifacts on your team website > valuation phase before your ARB for off-campus (DEN) students and instructional staff

20. Demos in ARB
For those teams doing a live demo in the ARB meeting, please include screenshots of your demo in your presentation
for your IV&Vers to see the demo in case video connection is a problem
for reviewers to make notes on

21. Skype/Google Hangout
Off-campus students who can not attend the ARB in-person, will be connecting through Skype or Google +
If your session is conducted during class time, you may use WebEx
Please finish setting up before the session starts

22. ARB timeslots
Monday Oct 15
Wed Oct 17
Fri Oct 19
12:30 – 1:50pm
SAL 322
2:00 – 3:20pm
OHE 122
4:00 – 5:20pm
3:30 – 4:50pm
5:20 – 6:40pm
5:00 – 6:20pm
Make reservation at
Clients and all team members, including off-campus students, must be available to attend

23. Past FCR Experiences and General Guidelines

24. Outline Previous FCR ARB Feedback Summary
Examples of Good and Bad Practices seen at ARBs
ARB Chartsmanship & Presentation

25. Overall FCR Feedback
Generally done well: presentations, time management, client rapport
Reconcile FCR content with ARB Success Criteria
When asked a question:
Give the answer in brief, this will help your time management and the Review Board will get the desired information
Listen Carefully, One speaker at a time
Many had poor time management that indicated that presentation(s) had NOT been practiced
Occasional pointing at laptop screen, not projected image (even better over Webex, use mouse)
Very occasionally, slides with NO value added 25 25

26. OCD Feedback (1) System boundary diagram
Generally done well: Organizational goals, Operational concept, System boundary and organizational environment.
Some benefits chains need rework:
Added stakeholders: users, clients, developers, IIV&Vers, database administrators, maintainers, interoperators, suppliers
Assumptions are about environment not about outcome
Involvement/use of system before system is built
Some organization goal(s) are Benefits Chain end outcomes
System boundary diagram
If you are using the component in/for your system, remove it from environment, e.g. PHP, .NET framework. 26 26

27. OCD Feedback (2) Organization Goals
Are Benefits Chain End Outcomes (or maybe a subset)
Are NOT project Initiative contributions
Identify Levels of Service properly
100% availability, 100% reliability - not feasible!
Make sure you can measure LOS goals
Prototypes and System are NOT the same (usually)
Business Workflow
Use activity-type diagram
Illustrate business activities
Not technical/system activity
May not even “see” system explicitly 27 27

28. Prototype Feedback
Generally done well: GUI Prototypes, Good understanding of client’s needs
Prototype all high-risk elements, not just GUI’s
COTS interoperability, performance, scalability
Use user/client-friendly terms
“John Doe, 22 Elm St.” not generic substitutions like “Name1, Addr1”
Use as an opportunity to gather more information and/or examples
Identify end users and try to get feedback from end users
Focus on important and high priority requirements (initially) 28 28

29. Requirements Feedback
Generally done well: Project and Capability requirements, OCD-requirements traceability
Prioritize all the requirements
Propagate LOS goals from OCD into SSRD or drop LOS requirements from Win Conditions (and SSAD)
Distinguish between client imposed requirements and developer choice solution (SSAD)
Make sure all requirements are testable
Qualify “24/7 Availability" with exceptions
Update the new requirements in WinBook tool
There is no such thing as an “implicit requirement” 29 29

30. SSAD Feedback Generally done well: Overall views
Follow UML conventions (arrows, annotations, etc.)
Generalization of actors
Uncommon mistakes in use-case diagrams
Two actors-one use case (means BOTH present)
Arrow direction for <extends> or <includes>
Devil is in the detail; simple is the best
Only two teams had an adequate start on Information & Arctifacts Diagram
Read the exit criteria for the milestone carefully 30 30

31. LCP Feedback - 1
Generally done well: overall strategy, roles and responsibilities
Too many 577b TBDs
Identify required skills for NN new team member(s) (577b; if needed to meet "team size")
Show (concentrate on) your future plan; not the past
Full Foundations phase plan
Don’t plan ONLY for documentation
Include Modeling
Include Prototyping; coding; executable architecture 31 31

32. LCP Feedback - 2 COCOMO drivers
Often differ per the module (type)
PMATs rationale was often wrong:
CS577 projects' process maturity should be between 2 and 3
Some driver rationales were "ridiculous"
Add DEN Student interactions to Gantt Chart
IIV&V
System/Project Engineer
Add maintainer’s responsibilities 32 32

33. FED Feedback
Generally done well: Business case framework, risk analysis
Specify LOS feasibility plans
Include training, operations, maintenance, opportunity costs/effort
Few had developers hours as cost (which is correct)
Try to quantify benefits, show return on investment
Change ROI to reflect on-going costs (possibly savings)
Distinguish one-time from annual costs in business case
Benefits start in mid 2017 (go at 6 months granularity); Costs start mid 2006
Elaborate process rationale 33 33

34. QFP Generally done well
Some missing traceability injection-removal matrix
Some seemed to try to "snow us with data", not present just a quick summary 34 34

35. Things to improve Presentation – communication skill
One word wrong could lead to billion $ loss.
Practice in front of others
Be concise and precise
Consistencies among each artifact
Team work vs. integrated individual works
Prepare your client:
Tell them what an ARB is (use agenda, success criteria)
Tell them what to expect from ARB
Time management
Get in and set-up ASAP
Have documents & client present 35 35