1. ilities Tradespace Workshop Summary Barry Boehm, Supannika Koolmanojwong USC-CSSE ARR 20 March 14, 2013 03-12-20131

2. Importance of ility Tradeoffs Major source of DoD system overruns System ilities have systemwide impact – System elements generally just have local impact ilities often exhibit asymptotic behavior – Watch out for the knee of the curve Best architecture is a discontinuous function of ility level – “Build it quickly, tune or fix it later” highly risky – Large system example below 03-12-20132

3. Importance of Cyber-Physical Systems Major gap in tradespace analysis capabilities Current ERS, DARPA tradespace research focused on physical system tradeoffs – Range, payload, size, weight, lethality, power and fuel consumption, communications bandwidth, etc. – Some focus on physical modularity, composability Current cyber tradespace research focused on software, computing, human factors tradeoffs – security, safety, interoperability, usability, flexibility, adaptability, dependability, response time, throughput, etc. Gaps in capabilities for co-design of hardware and software, integration of tradespace analyses 03-12-20133

4. iTAP Startup Results Top-level ility hierarchy view – Relation to JCIDS combat command user view – Draft survey to compare with acquirer, developer, supporter views Affordability means-ends framework view – Quantification via COCOMO, COSYSMO, CORADMO models Architecture-strategy synergies and conflicts views – Conflicts with other capabilities Change-Oriented Views: Incremental Commitment – The Cones of Uncertainty – MIT Epoch-Era Approach – Agile SE Schedule Acceleration Model Domain-Oriented Views – Ground: Wayne State, Georgia Tech – Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia 03-12-20134

5. SERC Value-Based ilities Hierarchy Based on ISO/IEC 9126, 25030; JCIDS; previous SERC research Individual ilities – Quality of Service: Performance, Accuracy, Usability, Scalability, Versatility – Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (size, weight, energy, …) – Protection: Safety, Security, Privacy – Robustness: Reliability, Availablilty, Maintainability – Flexibility: Modifiability, Tailorability/Extendability, Adaptability – Composability: Interoperability/Portability, Openness/Standards Compliance, Service-Orientation Composite ilities – Comprehensiveness/Suitability: all of the above – Dependability: Quality of Service, Protection, Robustness – Resilience: Protection, Robustness, Flexibility – Affordability: Quality of Service, Resource Utilization 03-12-20135

6. Prioritized JCIDS ilities User View by Combatant Commands: Top priority first Intelligence, Surveillance, and Reconnaissance – Comprehensive Persistent Survivable Integrated Timely Credible Adaptable Innovative Command and Control (note emphasis on Usability aspects) – Interoperability Understanding Timeliness Accessibility Simplicity Completeness Agility Accuracy Relevance Robustness Operational Trust Logistics: Supply – Responsiveness Sustainability Flexibility Survivability Attainability Economy Simplicity Logistics: Maintenance – Sustainability Responsiveness Attainability Flexibility Economy Survivability Simplicity Net-Centric: Information Transport – Accessible Capacity Accurate Timely Throughput Expeditionary Latency 03-12-20136

7. 7 Legacy System Repurposing Eliminate Tasks Eliminate Scrap, Rework Staffing, Incentivizing, Teambuilding Kaizen (continuous improvement) Work and Oversight Streamlining Collaboration Technology Early Risk and Defect Elimination Modularity Around Sources of Change Incremental, Evolutionary Development Risk-Based Prototyping Satisficing vs. Optimizing Performance Value-Based Capability Prioritization Composable Components,Services, COTS Affordability Improvements and Tradeoffs Get the Best from People Make Tasks More Efficient Simplify Products (KISS) Reuse Components Facilities, Support Services Tools and Automation Lean and Agile Methods Evidence-Based Decision Gates Domain Engineering and Architecture Task Automation Model-Based Product Generation Value-Based, Agile Process Maturity Means-Ends Framework: Affordability Reduce Operations, Support Costs Streamline Supply Chain Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based Tradeoffs and Balancing

8. 03-12-2013 8 USC: COCOMO II-Based Tradeoff Analysis Better, Cheaper, Faster: Pick Any Two? Slider-based equalizer version being developed -- Cost/Schedule/RELY: “pick any two” points (RELY, MTBF (hours)) For 100-KSLOC set of features Can “pick all three” with 77-KSLOC set of features

9. iTAP Startup Results Top-level ility hierarchy view – Relation to JCIDS combat command user view – Draft survey to compare with acquirer, developer, supporter views Affordability means-ends framework view – Quantification via COCOMO, COSYSMO, CORADMO models Architecture-strategy synergies and conflicts views – Conflicts with other capabilities Change-Oriented Views: Incremental Commitment – The Cones of Uncertainty – MIT Epoch-Era Approach – Agile SE Schedule Acceleration Model Domain-Oriented Views – Ground: Wayne State, Georgia Tech – Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia 03-12-20139

10. Architecture-Based Attribute Trades: Flexibility Example (RT-18a) Flexibility Arch. StrategySynergiesConflicts High module cohesion; Low module coupling Interoperability Reliability High Performance via Tight coupling Service-oriented architectureComposability, Usability, TestabilityHigh Performance via Tight coupling Autonomous adaptive systemsAffordability via task automation; Response time Excess autonomy reduces human Controllability Modularization around sources of change Interoperability, Usability, Reliability, Availability Extra time on critical path of Rapid Fielding Multi-layered architectureReliability, AvailabilityLower Performance due to layer traversal overhead Many built-in options, entry pointsFunctionality, AccessibilityReduced Usability via options proliferation; harder to Secure User programmabilityUsability, Mission EffectivenessFull programmability causes Reliability, Safety, Security risks Spare/expandable capacityPerformance, ReliabilityAdded cost Product line architecture, reusable components Cost, Schedule, ReliabilitySome loss of performance vs. optimized stovepipes 03-12-201310

11. MIT: ilities in Tradespace Exploration Based on Lean Aerospace, DARPA research Enabling Construct: Tradespace Networks Changeability Survivability Value Robustness Enabling Construct: Epochs and Eras Set of Metrics 03-12-201311

12. WSU: Versatility Factors and Physical Organization Components that Can be in Different Positions or Orientations Isolated or Separated Compartments Running Gear Chassis Turret SightWeapon suspension drive Mass & Structure Properties Mass Angular moments Imbalances* Load bearing wall strength Deck surface area Interior volumes** Interior surface areas** *Angular moments of the CG about axes of rotation ** By crew station and compartment 03-12-201312

13. Workshop objectives and approach Workshop objectives – Identify interested collaborators and data – Identify user needs for better reasoning about ility tradeoffs and affordability – Identify improved approaches for cyber-physical system co-design Workshop approach – Ask participants about their current and likely future challenges and research needs For ilities and their tradeoffs For cyber-physical co-design – Prioritize research with respect to strength of need, difficulty 03-12-201313

14. Workshop approach Ask participants about their current and likely future challenges and research needs – For -ilities and their tradeoffs – For cyber-physical co-design Prioritize research with respect to strength of need, difficulty

15. Participants Shawn Rahmani Gary Hafen Winsor Brown J.D. Baker Ed Colbert Thammanoon K. Peter Suk Rachchabhorn W. Sue K. Qi Li Lori Vaughan Qing Wang Jing Du Liming Zhu Da Yang Lee Osterweil Barry Boehm

16. Current and likely future challenges and research needs for -ilities and their tradeoffs Simulation model to address performance, usability model, comparing -ilities Define hierarchy, top factors Quantification of value of each –ilities, matrices Contribution of each –ilities (H/M/L. option1/2) Prototype for a short life vs requirements in terms of ilitlities Scale the tradeoff, scale up? Affordability (prioritizing issues, not only low cost) Inter-dependency of -ilities Pair-wise comparison tool (lockheed) Issue of subjective tradeoff study Modeling and trade study, matrices of architecture and –ilities Healthcare group (right distribution of data vs performance, privacy control/security) Usability vs design Parametric diagram Agile architecture development – tradeoff, architecture your asset & product How to specify requirements in a measurable way Security, accessibility, QoS Scalability – esp. diseconomy of scale, e.g. mobile network Reliability - 5 9s system in banking sector Macro view & micro view; complimentary process modeling (agent, task, resources) optimize the resources & others to optimize the choices, a tool that guide resource allocation (TWINS) Statistical mechanic to understand the process & relationship to COCOMO How to deploy, data /design selection; tradeoff guideline / tool for system implementation Architectural style tradeoff; compare & normalize & confidence level; validation tool &Sensitivity analysis tool to improve confidence level Fault tolerance; process view supporting tradeoff analysis How many (functional ) requirements is enough; size, # per release, customized development vs maintenance/enhancement unstated non-functional requirements Estimation – over/under-estimation Adaptability & flexibility – adapt to emergent behavior/requirements – not enough representative users How to help decision makers – can we perform as stated in the proposal Cyber vs physical; dual cone of uncertainty; Guidance /model ; how to do the trade/ how to weigh, checklist ; human process part Process for continuous delivery, how to prioritize requirements and assign them into each release based on revenue (considering dependency & process adoption) Using checklist – definition of Done(value, cost, quality) Scenario generators Product vs program characteristics - executability

17. Prioritization result Tools-ilities Models & Simulations (21) Multilevel Architecting (17) Prioritization (12) Metrics/ Matrices (8) Scenario generators (8) Checklists (7) Parametric diagrams (2) Affordability (14) Timeliness (12) Reliability / Fault Tolerance (11) Safety Scalability (10) Security / Privacy (10) Performance (5) Usability (4) Accessibility (3) Adaptability (3) Flexibility (1)

18. Afternoon Agenda Exploring high score -ilities factors – Exploring degree of difficulties Exploring high score tools Exploring cyber-physical co-design methods

19. Affordability Definition = Effectiveness & Cost (INCOSE) Fixed effectiveness & try to reduce cost Define cost (Total cost of ownership, life cycle cost) Cost = including operational & support cost? System cost (e.g. cheap to build to maintain but not cheap to operate) What is the biggest gap that is not covered when doing affordability analysis? – Technical debt As long as you are aware of your debt, give you leeway Technical debt identification tool – HW SW integration

20. Timeliness Define timeliness – Meet deadline subject to definition of effectiveness (timebox – discard low priority items to meet schedule; agile rebaselining) – Version control issues – Conflict in reusing items

21. Reliability / Fault Tolerance /Safety Standard assurance issue Define “safe” Reliability = impact of defect (loss of life/properties/ $) Dependent on other systems – Cloud – no full control How can your system tolerate other systems? Not equal (reliable <> fault tolerance <> safety) Acceptable levels A tool to provide analysis of balancing these factors KPP – key performance parameter – no single number Estimating certification cost & schedule Level of testing

22. Scalability Scalability of Product Architecture evaluation tool – Something like static source code analysis tool Modeling tool Network traffic & overhead Communication mechanism Co-dependent systems (with internet connection? ) Diversity – Versions or platforms Horizontal vs Vertical (scale out vs scale up) Consistency among diversity/nodes Timing How to upgrade

23. Security / Privacy Acceptable levels / level of assurance Security vs complexity tradeoff (also with reliability, availability) multiple independent level of security Certification cost & schedule Scope of certification (privacy) Lifetime of data (dynamic data) Anti-tamper communication security Scaling How to show feasibility evidence How to justify being assured (how the entire thing is developed) Proof of correctness Domain dependent

24. Prioritization result Tools-ilities Models & Simulations (21) Multilevel Architecting (17) Prioritization (12) Metrics/ Matrices (8) Scenario generators (8) Checklists (7) Parametric diagrams (2) Affordability (14) Timeliness (12) Reliability / Fault Tolerance (11) Safety Scalability (10) Security / Privacy (10) Performance (5) Usability (4) Accessibility (3) Adaptability (3) Flexibility (1)

25. Models & Simulations (Multilevel) Accuracy / fidelity VV&A (verification, Validation & accreditation) Scalability Modeling environment (jungle/desert) Modeling language Visualization Test cases, usage(scope of use), underlying assumption

26. Architecting Language (dependent) Visualization (static & dynamic) Executable architecture Analyzable Multiple views (like DODAF), integration of views Generation of the system from the architecture Architecture pattern, reuse

27. Prioritization Multiple stakeholder value proposition, criteria analysis Tradeoff Interdependencies Cost & schedule Value estimation Environment (scenario)