1. CSW ToK Meeting III (M4) Presentation Nuno Silva Critical Software S.A. nsilva@criticalsoftware.com Naples, 25th January 2011

2. N APLES, 25 TH J AN. 2011 CSW M4 P RESENTATION [1] ::.. Agenda Recruitment Status Secondments Status WP2 status Next Secondement activities

3. CSW M4 P RESENTATION [2] N APLES, 25 TH J AN. 2011 ::.. Recruitment Status -2 researchers have been contracted: -As’ad Salkham, PhD -Marcin Karpinski, PhD -Issues with HR are resolved (contract, CSW apartment, computers); -Researchers are installed in Coimbra, already found apartment, etc; -Researchers are already involved in projects: pShield: pilot embedded Systems arcHItecturE for multi-Layer Dependable solutions / ISVV contributions -D1.6 Researchers recruited document produced

4. CSW M4 P RESENTATION [3] N APLES, 25 TH J AN. 2011 ::.. Recruitment Status -Contract is from July 2010 to May 2012 -Lessons Learned: -Many “unforeseen” risks have arised (position publication, time for VISA, etc) – exhaustive list of risks must be planned for next projects -Budget calculations must be clarified. From the total amout paid to the researchers we had to do many simulations according to Portuguese Labour laws (Social Security, income taxes, work insurances, company rules – working time, vacation periods, etc) -Do not forget that some company related costs are applied (CSW Overheads with HR, location, services, etc) -Support time is not negligible

5. CSW M4 P RESENTATION [4] N APLES, 25 TH J AN. 2011 ::.. Recruitment Status -Lessons Learned (cont’d): -Depeding on external consulting companies for contracts and labour law was not a positive experience. It is preferable to plan ahead of time for someone to do the research with internal HR department (this was the way we clarified all questions).

6. CSW M4 P RESENTATION [5] N APLES, 25 TH J AN. 2011 ::.. Secondments Status

7. CSW M4 P RESENTATION [6] N APLES, 25 TH J AN. 2011 ::.. Secondments Status CSW “experts” 2010: Nuno Silva @ FCTUC - July (2 weeks) – Done - August/September (2 weeks) - Done - December (2 weeks) - Done Ricardo Barbosa @ CINI/UoN - June/July (1 month) - Done -October/November (2 months) – Done -November 2011 (1 month) - started on 24/1 Nuno Silva @ FCTUC - January/February 2011 (1 month) – started on 17/1

8. CSW M4 P RESENTATION [7] N APLES, 25 TH J AN. 2011 ::.. Secondments Status Other “experts” 2010: Henrique Madeira @ CSW - May (2 weeks) – Done Marco Vieira @ CSW - May (2 weeks) – Done João Durães @ CSW - December (2 weeks) – Date to be agreed (February 2011)

9. CSW M4 P RESENTATION [8] N APLES, 25 TH J AN. 2011 ::.. Secondments Status Other “experts” 2010: Stefano Russo @ CSW - Feb/Mar/Sept (4 weeks) – Done Domenico Cotroneo @ CSW - Feb/Mar/Sept (4 weeks) – Done Christian Esposito @ CSW - July/October (12 weeks) – Done Catello Di Martino @ CSW - September/October (6 weeks) – Done

10. CSW M4 P RESENTATION [9] N APLES, 25 TH J AN. 2011 ::.. Secondments Status Other “experts” 2011: Stefano Russo @ CSW - Jan/Feb (4 weeks) – to be started in February Domenico Cotroneo @ CSW - April (4 weeks) – to be anticipated/started next week - August (4 weeks) – to be planned Christian Esposito @ CSW - February to July (5 months) – to be started next week Catello Di Martino @ CSW - 3 months secondement – to be replaced

11. N APLES, 25 TH J AN. 2011 CSW M4 P RESENTATION [10] ::.. WP2 status WP1 Project Management Leaded by CINI/UoN Input parameters for evaluation Techniques and Mechanisms Results Techniques WP2 Qualitative eval. of OTS items Leaded by CSW WP5 Prototype tools design Leaded by SESM WP3 Quantitative evaluation of robust. and depend. Leaded by FCTUC WP4 Techniques for on-line fault diagnosis and dynamic reconfiguration Leaded by CINI/ UoN CSW was responsible for WP2 and contributes to all other WPs

12. N APLES, 25 TH J AN. 2011 CSW M4 P RESENTATION [11] ::.. WP2 status WP1 Project Management Leaded by CINI/UoN WP2 Qualitative eval. of OTS items Leaded by CSW WP2 focus on definition of metrics for the qualitative evaluation of OTS items to be integrated in Large- scale Complex Critical Infrastructures (LCCI) software.

13. N APLES, 25 TH J AN. 2011 CSW M4 P RESENTATION [12] ::.. WP2 status WP L EADER TASKD ELIVERABLES M ONTH P ARTNER WP 2 Q UALITATIVE EVALUATION OF OTS ITEMS CSW Nuno Silva T2.1 Criteria for qualitative evaluation Task Leader: Ricardo Barbosa D2.1: Requirements for OTS items evaluation (report) Jun 2010 CINI/UoN FCTUC D2.2: Progress report and theoretical frame- work on OTS items classification and selection Aug 2010 T2.2 OTS items classification and selection Task Leader: Ricardo Barbosa D2.3: Guidelines to support software engineers during the OTS selection phase (for dissemination purposes) Dec 2010 T2.1 and T2.2 have been completed: D2.1 has been finalised, reviewed and published D2.2 has been finalised, reviewed and published D2.3 has been finalised and reviewed – no comments have yet been received – to publish.

14. CSW M4 P RESENTATION [13] N APLES, 25 TH J AN. 2011 ::.. D2.1 Requirements for OTS items evaluation Document overview 2.Off-The-Shelf Project Domain 2.1State of the art in OTS evaluations 2.1.1Functional testing 2.1.2Robustness testing 2.1.3Stress testing 2.1.4Reliability testing 2.1.5Dependability benchmarks 2.2Related Research Projects 2.2.1The European Space Agency (ESA) Ravenscar Benchmark (ERB) 2.2.2Space and Naval Warfare Systems Center COTS Evaluation Process 2.2.3COTS-Based Software Development: Processes and Open Issues 2.2.4Techniques for COTS Software Evaluation 2.2.5DesCOTS

15. CSW M4 P RESENTATION [14] N APLES, 25 TH J AN. 2011 ::.. D2.1 Requirements for OTS items evaluation 2.2.6OPAL 2.2.7Others similar works 2.3Common applications for OTS within commercial systems 2.3.1Avionics 2.3.2Automotive 2.3.3Space 2.3.4Defence Systems 2.3.5Air Traffic Management 2.3.6Web Applications 2.4Common Types of OTS 2.4.1Generic OTS Components 2.4.2Some Mission-Critical Specific OTS libraries 2.4.3OTS in Safety Critical Systems 2.4.4Common Real Time Operating Systems 2.4.5Common Toolsets used for Verification and Validation

16. CSW M4 P RESENTATION [15] N APLES, 25 TH J AN. 2011 ::.. D2.1 Requirements for OTS items evaluation 3.Requirements for OTS items evaluation 3.1Requirements Driven from Standards 3.1.1ISO/IEC 14598 3.1.2ISO/IEC 9126 3.1.3ISO/IEC 25051 3.1.4ISO/IEC 60880 3.1.5EN 50128 3.1.6DO-178B 3.1.7NASA-STD-8739.8 3.1.8ECSS-Q-80

17. CSW M4 P RESENTATION [16] N APLES, 25 TH J AN. 2011 ::.. D2.1 Requirements for OTS items evaluation 3.2Requirements Driven from Guidebooks and Reports 3.2.1Software Product Evaluation and Certification 3.2.2Federal Aviation Administration 3.2.3Off-The-Shelf Option 3.2.4Social-Technical Approach to COTS Evaluation 3.2.5Internal Projects 4.Conclusions 4.1Reusable documents and standards Annex A.Summary of Requirements for Evaluation of OTS Components

18. CSW M4 P RESENTATION [17] N APLES, 25 TH J AN. 2011 ::.. D2.2 Progress report and theoretical frame-work on OTS items classification and selection Document overview 2.Framework definition 2.1Overview 2.2Introductory concepts 2.3The Quality Model concept 2.4Framework Processes 3.Decision algorithms 3.1Overview 3.2Multi Criteria Decision Making (MCDM) Algorithms 3.3Analytical Hierarchy Process (AHP) 3.4Weighted Sum Model (WSM) 4.Conclusions

19. CSW M4 P RESENTATION [18] N APLES, 25 TH J AN. 2011 ::.. D2.2 Progress report and theoretical frame-work on OTS items classification and selection

20. CSW M4 P RESENTATION [19] N APLES, 25 TH J AN. 2011 ::.. D2.3 Guidelines to support software engineers during the OTS selection phase This deliverable aims to provide a sort of guideline for researchers and practitioners in the field of software systems design and development for LCCIs, as well as for SCSs. It will was edited in the form of a manual, aiming to give useful hints and suggestions even in terms of examples coming from real world case studies. Delivered a few weeks ago for comments – is going to be publish on the CSTEP web-site soon.

21. CSW M4 P RESENTATION [20] N APLES, 25 TH J AN. 2011 ::.. D2.3 Guidelines to support software engineers during the OTS selection phase 1.Introduction 2.Guidelines for the OTS selection 2.1.Overview 2.2.Framework Defined Processes 2.3.Guidelines to the proposed process 2.3.1.Quality Model Definition 2.3.1.1.sub-domain definition 2.3.1.2.Properties definition 2.3.1.3.Metrics definition 2.3.1.4.Criticality levels metrics target values definition

22. CSW M4 P RESENTATION [21] N APLES, 25 TH J AN. 2011 ::.. D2.3 Guidelines to support software engineers during the OTS selection phase 2.3.2.Evaluation Definition and Execution 2.3.2.1.Quality Model selection 2.3.2.2.OTS (and types) definition and selection 2.3.2.3.Methods/tools categories and configurations definition 2.3.2.4.Evaluation execution 2.3.3.Qualification Assertions 2.3.3.1.Operation instructions (tasks) 2.3.3.2.Qualification Scheme selection and configuration 2.3.3.3.Quality Model and Criticality Level selection 2.3.3.4.Evaluation(s) selection 2.3.3.5.Qualification Campaign execution 3.Results interpretation from using the framework 3.1.Classification and selection Annex A.Base Quality Model

23. CSW M4 P RESENTATION [22] N APLES, 25 TH J AN. 2011 ::.. Next Secondment Activities As much as possible we will try to have industrial involvment in COTS evaluation (robustness and dependability) and new techniques definition for fault diagnosis. WP1 Project Management Leaded by CINI/UoN Input parameters for evaluation Techniques and Mechanisms Results Techniques WP2 Qualitative eval. of OTS items Leaded by CSW WP5 Prototype tools design Leaded by SESM WP3 Quantitative evaluation of robust. and depend. Leaded by FCTUC WP4 Techniques for on-line fault diagnosis and dynamic reconfiguration Leaded by CINI/ UoN

24. CSW M4 P RESENTATION [23] N APLES, 25 TH J AN. 2011 ::.. Next Secondment Activities ?

25. CSW M4 P RESENTATION [24] N APLES, 25 TH J AN. 2011 ::.. Next Secondment Activities So far, the targetted projects/types of projects are: - csXception re-architecture and implementation of an ARM fault injection plug-in for automotive industry - Support csXception architecture review and suggest modification for supporting LCCI and middleware fault models - Definition of generic fault model for LCCI / specific Middleware faults and how to measure the results - Application of the plug-ins in a cases study (from CSW or other partner) - Support to development of V&V “Bible”

26. CSW M4 P RESENTATION [25] N APLES, 25 TH J AN. 2011 ::.. Next Secondment Activities Targetted projects/types of projects (Cont’d): - Review of CSW process for support to certification of SW/Systems, namely applied to on-going projects - Support DO-178B activities, but also automotive industry V&V and Certification - These tasks are heavily dynamic and dependent on customer needs/availability and information level of disclosure - Mainly Robustness tests strategy and methodology definition and execution, and dependability full evaluations of the systems - Support to product (COTS) qualification activities

27. CSW M4 P RESENTATION [26] N APLES, 25 TH J AN. 2011 ::.. Next Secondment Activities WP3 – Quantitative evaluation... - csXception related work - Robustness/Dependability processes implementation and metrics extraction - Safety Validation process - Safety Cases support WP4 – Techniques for on-line fault diagnosis... -Result of application of the the previous techniques - ISVV type of recommendations - Automotive industry is heavily dependent on fault diagnosis and currently growing market for CSW

28. CSW M4 P RESENTATION [27] N APLES, 25 TH J AN. 2011 Questions? Comments? Thank you! nsilva@criticalsoftware.com