© 2014 Noblis, Inc. Noblis proprietary and confidential. Using Systems Engineering Processes and Methods to Make Procurements Easier and to Fully Meet Public Transportation Agency Expectations Blake Christie and Ann Diephaus April 3, 2014

22 © 2014 Noblis, Inc. Noblis proprietary and confidential. The Transportation Challenge  More Vehicles  More Drivers  More Trips But –  Fewer Resources  Space Limits on New Road Construction  Frequent Local Opposition to Mass Transit Solutions Means – We have congestion, long commutes, dangerous roadways, and …

33 © 2014 Noblis, Inc. Noblis proprietary and confidential. So What’s The Solution?  More money would help, but that’s not the total answer  Technology can help – Intelligent Transportation Systems  Intelligent Transportation Systems (ITS) include: Better information for the surface transportation system manager (e.g., sensors, cameras, weather stations) Better information for the surface transportation system user (e.g., 5-1-1, travel time maps on the Internet) Technology that reduces the time spent in travel-related activities, through the use of automation (e.g., toll collection)

44 © 2014 Noblis, Inc. Noblis proprietary and confidential. Technology Solutions Bring Their Own Challenges  New technology is generally proprietary – and expensive  Competing technologies don’t start out interoperating, complicating the use of similar items in different locations (e.g., Smart Tag vs. E-Pass)  Users don’t necessarily have a “vision” for how to integrate all of the necessary systems In transportation, engineers are primarily Civil Engineers, experienced in construction, not systems Information systems involve software, which has its own known set of challenges In the early days of ITS (and to a lesser degree today), there was the “NIH” syndrome

55 © 2014 Noblis, Inc. Noblis proprietary and confidential. U.S. Department of Transportation Role  Fostered research and development of: “Intelligent Highways” Intelligent Transportation Systems (successor to “intelligent highways”)  Sponsored the development of a National ITS Architecture Conceptual description of the expected systems and the infrastructure that ties them together Framework for states and other jurisdictions building intelligent transportation systems Adding connected vehicles architecture Provides the “Big Picture”  Recognized the need for standards to support the development of intelligent transportation systems and sponsored their development

66 © 2014 Noblis, Inc. Noblis proprietary and confidential. Issues With ITS Standards Development  Early standards weren’t successful Failed to capture the full scope of the areas they were intended to address Were limited by the lack of systems experience by the people developing them Were ignored (because of limitations) by the agencies fielding ITS systems Were hard to read and ambiguous  Revisions to standards weren’t successful Groups working on the standards lacked sufficient systems expertise Users were inadequately involved in modifications, largely because of a lack of systems expertise – but the standards were intended for them!!

77 © 2014 Noblis, Inc. Noblis proprietary and confidential. Systems Engineering – To the Rescue!  Proposal made to use a systems engineering process for ITS standards development  Why? Provide a context for the standard – concept of how it would be used in actual operation of a system Develop clear-cut requirements, based on user needs, for the interfaces and devices requiring standardization Trace the requirements back to user needs, to show users how the standard evolved Design standard solutions that addressed requirements Trace standard solutions back to requirements Create mechanism for testing products that claimed conformance to the standard

88 © 2014 Noblis, Inc. Noblis proprietary and confidential. Development of the Systems Engineering Profile for ITS Standards  Identified the appropriate “ilities” Usability Readability Maintainability Interoperability Flexibility  Mapped the interface standards development process to system life-cycle stages Concept of Operations Requirements Needs-to-Requirements Matrix Design Requirements Traceability Matrix Verification and Validation

9 © 2014 Noblis, Inc. Noblis proprietary and confidential. 9 The “V” Diagram (Development Life-Cycle) Concept of Operations High Level Requirements Detailed Requirements High Level Design Detailed Design Implementation Operations & Maintenance System Acceptance Subsystem Verification Integration & Test Time Relates to Configuration Control Begins With the First Product and Continues Throughout the Life of the System Development Translates Design Into Product

10 © 2014 Noblis, Inc. Noblis proprietary and confidential. Systems Engineering and Quality  Quality Concerns Exist: Within each stage At the transitions between stages  Within a stage, solutions must validly reflect the intent and content of the product(s) of the previous stage  At the transitions, the stakeholders must verify the products that result from a stage Each product must be complete Each product must be correct Concept of Operations High Level Requirements Detailed Requirements High Level Design Detailed Design Implementation Operations & Maintenance System Acceptance Subsystem Verification Integration & Test We want to improve the quality of the standards

11 © 2014 Noblis, Inc. Noblis proprietary and confidential. What Systems Engineering Provides to ITS Standards Development  Creates a quality standard  Ensures that the standard meets user needs  Verifies that the standard is complete  Validates that the standard is correct  Supports deployments  Reduces cost of interface development

12 © 2014 Noblis, Inc. Noblis proprietary and confidential. Profile for a ConOps for a Standard  Define User Needs Description of what the users want to do Highest level of “requirement” in the system  Establish Operational Policies and Constraints What policies govern the operation of the system What constraints does the system have to accommodate  Delineate Modes of Operation Normal mode (rush hour, scheduled events, etc.) Other modes (accidents, sever weather, tec.)  Provide Operational Scenarios (optional) – used to give examples of how the user (or system) may operate with the capability desired  Tell a story  Introduced use of Guides: IEEE Std (ConOps) INCOSE SYSTEMS ENGINEERING HANDBOOK, version 3.1, section IEEE Std (for technical reviews and walkthroughs)

13 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced the Characteristics of Good Functional Requirements  Necessary  Concise (minimal, understandable)  Attainable (achievable or feasible)  Complete (standalone)  Consistent  Unambiguous  Verifiable Guided by IEEE & INCOSE SYSTEMS ENGINEERING HANDBOOK version 3.1, Appendix I Needs Requirements Based Needs to Requirements Matrix

14 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced the Profile of Design Concepts  Based on the requirements  Directly traceable to one or more requirements  Consistent with the requirements  Follow rule of one (a single) definition (How many times should you define an object?) Requirements Designs Driven Requirements Traceability Matrix (RTM)

15 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced Basics of Verification and Validation Verification and Validation are related concepts  Verification is “building the product right” -- ensures that all functions implemented in the product have been implemented correctly  Validation is “building the right product” – ensures that the desired functions have been implemented in the delivered product V&V Methodologies

16 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced What V&V is at Each Stage Concept of Operations High Level Requirements Detailed Requirements High Level Design Detailed Design Implementation Operations & Maintenance System Acceptance Subsystem Verification Integration & Test V&V for ConOps asks, is this a complete set of needs and is each need correctly described? V&V for requirements asks, do the requirements address all the needs (completeness) and do respective requirements fulfill the need (correctness)? V&V for the design asks, are the requirement(s) all addressed (completeness) and does the design fulfill each requirement (correctness)?

17 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced the Needs-to-Requirements Matrix Shows which requirements fulfill specific user needs  Every Need Must Be Addressed By At Least One Requirement  Every Requirement Must Relate to At Least One Need  Any Need That is Not Addressed By At Least One Requirement may be Irrelevant  Every Requirement That Does Not Address At Least One Need is Irrelevant

18 © 2014 Noblis, Inc. Noblis proprietary and confidential. What is a Needs-to-Requirements Matrix used for?  Provides tool for completeness and correctness (V&V)  Helps guide procurements  Establishes high level picture (how it fits)  Maps (references) to details  Sets up basis for design

19 © 2014 Noblis, Inc. Noblis proprietary and confidential. What Does the Needs-to-Requirements matrix Look like in ITS Standards User Need ID User Need FR ID Functional Requirement ConformanceSupportAdditional Specifications Define a MessageVMS:MYes / NA Determine Maximum Number of Pages MYes Determine Maximum Message Length MYes Determine Supported Color Schemes MYes Configure Default Flash-On and Flash- Off Times OYes / No The DMS shall support all flash on times from ____ tenths of a second (0..255) to ____ tenths of a second (0..255) in ____ tenths of a second increments. The DMS shall support all flash off times from ____ tenths of a second (0..255) to ____ tenths of a second (0..255) in ____ tenths of a second increments.

20 © 2014 Noblis, Inc. Noblis proprietary and confidential. How is the Needs-to-Requirements Matrix Used to make Procurements Easier?  The system owner selects the operational concepts they need  The system owner addressed the optional requirements as needed  The filled in needs-to-requirements matrix becomes the requirements selected for the device interface  When included “as specified in the standard” in contract language, off-the- shelf implementations are achievable  Basis for all interface testing “Overall, VTTI feels that the project has been a resounding success. The specification process meets the goals of creating a more user-friendly environment for an agency to develop procurements.” Ashwin Amanna Virginia Tech Transportation Institute Deployment and Testing of an Updated Dynamic Message Sign (DMS) Standard, FINAL REPORT, April 24, 2007

21 © 2014 Noblis, Inc. Noblis proprietary and confidential. What Does the Needs-to-Requirements matrix Look like in ITS Standards User Need ID User Need FR ID Functional Requirement ConformanceSupportAdditional Specifications Define a MessageVMS:MYes / NA Determine Maximum Number of Pages MYes Determine Maximum Message Length MYes Determine Supported Color Schemes MYes Configure Default Flash-On and Flash- Off Times OYes / No The DMS shall support all flash on times from ____ tenths of a second (0..255) to ____ tenths of a second (0..255) in ____ tenths of a second increments. The DMS shall support all flash off times from ____ tenths of a second (0..255) to ____ tenths of a second (0..255) in ____ tenths of a second increments.

22 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced the Requirements Traceability Matrix (RTM)  Tells developers how specifically to meet a given requirement  Traces from requirements to design (dialog and associated objects) Every Requirement Must Be Addressed By At Least One “Thing” (Dialog, Object, Element) Every “Thing” Must Relate Back to At Least One Requirement Any Requirement That is Not Addressed By At Least One “Thing” is Unfulfilled (Unsatisfied) Any “Thing” That Does Not Address At Least One Requirement is Unnecessary

23 © 2014 Noblis, Inc. Noblis proprietary and confidential. What is a RTM Used For?  Provides tool for completeness and correctness checks (V&V) of the design  Specifies the “how” for development  Maps (or references) requirements to design

24 © 2014 Noblis, Inc. Noblis proprietary and confidential. What does the RTM Look like in ITS Standards? FR IDFunctional Requirement Dialog ID Object ID Object Configure Default Flash- On and Flash-Off Times G defaultFlashOn 5.5.5defaultFlashOff

25 © 2014 Noblis, Inc. Noblis proprietary and confidential. Slide-25 Example Dialog CenterSignal Controller GET (activeVolumeOccupancyDetectors) RESPONSE (activeVolumeOccupancyDetectors) GET (volumeOccupancyTable) RESPONSE (volumeOccupancyTable) Determine number of rows in table GET (volumeOccupancySequence) RESPONSE (volumeOccupancySequence) Determine if a new table is available Collect volume & occupancy data Read Volume/Occupancy Data Dialogue Sequence Objects

26 © 2014 Noblis, Inc. Noblis proprietary and confidential. How is the RTM used to make procurements and testing easier?  The RTM specifies how the interface is to precisely behave  Think of it as an Interface Control Document (ICD)  When included “as specified in the standard” in contract language, off-the- shelf implementations are achievable  Basis for all interface testing

27 © 2014 Noblis, Inc. Noblis proprietary and confidential. 27 Introduction of Advanced Verification Concepts Concept of Operations High Level Requirements Detailed Requirements High Level Design Detailed Design Implementation Operations & Maintenance System Acceptance Subsystem Verification Integration & Test Time Relates to Advanced Planning of Verification (Inspection, Analysis, Demonstration, Test) Methods Introduced to work on the right side of the “Vee”

28 © 2014 Noblis, Inc. Noblis proprietary and confidential. Introduced standardized test procedures Problem: Planning for verification was limited, resulting in inconsistent testing for conformance Solution:  Provides a requirements to test case/test procedure matrix  Provides IEEE std guided test case/test procedures  Standardized test procedures verify conformance  Implementations use the test procedures; results are consistent “… The testing team was able to quickly identify problems and assign responsibility. This process fostered an amicable environment between the agency and the vendor which produced very fast resolution to problems.” Ashwin Amanna Virginia Tech Transportation Institute Deployment and Testing of an Updated Dynamic Message Sign (DMS) Standard, FINAL REPORT, April 24, 2007

29 © 2014 Noblis, Inc. Noblis proprietary and confidential. Requirements to Test Case/Procedure Martix By planning the testing out, local agencies were assured that the system was verified. Requirements to Test Case & Test Procedure Matrix RequirementTest Case IDTitleIDTitle Execute Climate-Control Equipment Testing C C Climate-Control Equipment Test - No Errors Climate-Control Equipment Test - Errors

30 © 2014 Noblis, Inc. Noblis proprietary and confidential. Standardized test procedures Test Case & Test Procedure Consistent Formats StepTest ProcedureResults Additional References 1CONFIGURE: Determine the event class to utilize for this test (e.g., per the test plan). RECORD this information as: »Class_Index 16GET the following object(s): »eventConfigStatus.Event_Index Pass / Fail (Section ) Section H Step d 17VERIFY that the RESPONSE VALUE for eventConfigStatus.Event_Index is equal to ‘log’ (3). NOTE--Valid enumerated values for eventConfigMode are defined in NTCIP 1103, Section A (Event Log Configuration Status Parameter). Pass/Fail (Section ) Includes test descriptions, identification of variables, test case results, etc.

31 © 2014 Noblis, Inc. Noblis proprietary and confidential. Summary Systems engineering  Formalizes ITS Standard development in three stages (ConOps, requirements, design)  Ensures quality in deployable standards that support interoperability and interchangeability  Provides for Verification & Validation of the standard at each stage  Combines systems engineering expertise with existing domain expertise, as best practices recommend The new ITS Standards, developed using systems engineering concepts make it easier to procure and test

32 © 2014 Noblis, Inc. Noblis proprietary and confidential. Questions?

33 © 2014 Noblis, Inc. Noblis proprietary and confidential. Additional References  Applying Systems Engineering Principles to the Development of Transportation Communication Standards - See more at: #sthash.dgcraB3G.dpuf #sthash.dgcraB3G.dpuf  The systems engineering process is used during the development of ITS standards - See more at: #sthash.yj2xUrat.dpuf #sthash.yj2xUrat.dpuf  The ITS Standards Program is managed and funded by the United States Department of Transportation (USDOT), Intelligent Transportation Systems Joint Program Office (ITS JPO), Steve Sill is the Program Manager and can be contacted at or  USDOT ITS Standards Website  For further information, contact: Blake Christie at and Ann Diephaus at and