Steven Arndt, Ph.D., P.E. Chairman Maryland State Board for Professional Engineering September 16, 2015

Presentation Overview State Board for Professional Engineering Introduction to “catastrophic” software failures or do software developer really need to be licensed Some catastrophic software failures How can licensing software engineers help? The development of the Software PE exam Update Questions 2

State Board for Professional Engineering Qualifies and licenses persons seeking licensure to practice as a professional engineer The Board regulates the practice of engineering under the provisions of Business Occupations and Professions Article, Annotated Code of Maryland, Title 14 and the Board investigates complaints against licensees, as well as complaints related to unlicensed practice. The Board may issue a reprimand, suspend or revoke a license and /or impose a fine 3

Which systems affect the health, safety and welfare of the public? Typical domains – medicine, transportation, infrastructure, commerce, finance Typical applications – implantable medical devices, automobiles, elevators, power systems, financial and health record management systems Others – entertainment – e.g. amusement park ride – consumer goods – e.g. microwave oven – … etc. 4

Scenario Consider a closed loop insulin pump / artificial pancreas reads blood glucose doses insulin automatically A software failure causes overdoses 7 patients die before product is recalled Who is responsible? What happens next? 5

Real world Between 2005 and 2009 the FDA received ~ 56,000 reports of infusion pumps failures numerous cases of injury and death 87 infusion pumps recalled 14 recalled due to “Class I” safety concerns devices could cause serious adverse health consequences or death. Software defects lead to over-and-under infusion alarms on pumps to fail in emergencies or activate in absence of a problem FDA offers manufacturers pre-market review of software code used in infusion pumps 6

Famous Software Catastrophes Therac 25 ( ) a radiation therapy device malfunctions and delivers lethal radiation doses to 6 patients Explosion of Ariane 5 rocket (1996) Loss of more that $350 million Caused by a common cause software exception (arithmetic overflow) Patriot Missile Battery Intercept Failure Death of 28 soldiers and injury of 98 Caused by “software aging” Many so-called software failures are often found to be Human error Mechanical failure Environmentally caused Some combination 7

Another Disaster November National Cancer Institute, Panama City A therapy planning program developed by a US firm miscalculates radiation therapy dosages Dosage errors caused by doctors misusing program doctors try to “trick” software into making calculations it was not intended to do software miscalculates and doubles dosages failure to manually verify treatment plan At least eight patients die 20 receive overdoses likely to cause significant health problems Three radiological technicians are indicted for murder – two found guilty What was the culpability of the software/software engineers? 8

Another Scenario 110 million Target customers have had their personal data stolen in a data breach Stolen information included credit and debit card data customer names and PIN (personal identification data) numbers One in three Americans affected Is this catastrophic? Who is to blame? 9

Critical IT Infrastructures telecommunication infrastructure water supply electrical power system oil and gas road transportation railway transportation air transportation banking and financial services public safety services healthcare system administration and public services 10

Have there really been many failures? Just one major organization list thousands of software failures 1985-present voluntarily reported failures whistleblowers examining corporate documents newspaper stories lawsuits privately settled cases government announcements Many are of critical infrastructures Some include injury and loss of life 11

Liability Perspective Most computer-related transactions are considered a sale of "goods" Are covered by Uniform Commercial Code warranties, terms of the contract, preempt negligence claims Software agreements include exculpatory language integration clauses warranty disclaimers limits on remedies to the repair or replacement of defects Shifts the risk of software failure from the seller to the user How do we protect the public from harm from software failures? 12

Is Software Engineering Really Engineering ? Engineering is the creative application of scientific principles to design or develop “something” Software has standards for design, construction, test, maintenance, etc. Software engineering includes metrics and models for measuring failure rate, failure consequence, etc. Software analyses methodologies help identify problems before they appear in the real environment (should be used for designing, testing and coding) 13

Why is Software Engineering so Important Today? Engineered systems are becoming software intensive: flight systems, financial systems, defense systems, energy related systems, etc. Software intensive systems are often safety critical: air traffic control systems or medical systems, etc. Cost of developing software is increasing The share of software in systems is important, its cost becomes a major contributor to the cost of the project. The cost of software is essentially associated with corrective actions. It varies between 40 and 70% 14

Why licensure? States require licensure of certain engineers to ensure that any practitioner is at least minimally competent Intent is to protect the public from injurious consequences of incompetent “engineers” Licensure is required if the engineer is involved in building a system – whose failure could cause significant harm – is offering his services directly to the public – and not through a corporation, or government entity 15

How does licensing software engineers help? Raising the level of professionalism Creating a system of accountability Creating trust and confidence in the public Qualifying expert witnesses But …. only a small percentage of software professionals will need to be licensed my thoughts on this… 16

Licensure requirements Bachelor’s degree in software engineering from an ABET- accredited program (21 of them in US) Fundamentals of Engineering (FE) exam Applicable, supervised work experience (typically, at least four years) Principles and Practices of Software Engineering (P&P) exam Evidence of good moral character Continuing education to retain licensure 17

History of Software PE Exam In Texas, the licensing board for professional engineers began offering licensure to software engineers in 1999 There was no standardized exam covering software engineering in place. In 2006, the board changed its rules to require all applicants for licensure to pass a PE exam This change effectively cut off the path to licensure for software engineers in the state Texas board believes that software engineering is a critical component of many engineering projects and that it’s important to recognize the impact that software engineering has on the public’s health, safety, and welfare 18

Examination Development Process Overview Equating of Examination Item Writing & Review Examination Assembly & Review Examination Scoring Standard Setting Study Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed Examination Administration After Cut Score Established 20

Examination Development Process Overview Need for Examination Identified 21

Need for Examination or Module Identified Request by no fewer than 10 Member Boards Proof of need Estimate of usage Impact on health, safety, and welfare Is not adequately tested by an existing exam Must be at least one EAC/ABET program Initially in partnership with a technical society 22

History of Software PE Exam In August 2009 the NCEES approved the development of a new PE exam for software engineers 10 NCEES member boards presented letters of support IEEE-USA, IEEE Computer Society, NSPE and the Texas Board sponsored the development of the exam Sufficient ABET programs were identified The first exam was given in April of 2013 Professional actives and knowledge study (PAKS) Body of knowledge development / Examination Specification Exam questions 22

Examination Development Process Overview Need for Examination Identified Task Analysis Performed (PAKS) 24

Professional Activities and Knowledge Study (PAKS) Procedure Committee structure Survey structure A list of tasks, knowledge, and skills that committee feels may be important to the safe practice of their profession at the time of licensure Survey generated based on the task statements and important knowledge and skills from above and will be rated by survey respondents 25

Examination Development Process Overview Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed 25

Specification Development Specification Development Committee Based on analysis report Identification of knowledge areas Passing list (2.5 and above) Borderline list (2.4 to 2.5), may be included if strong rationale Failing list (less than 2.4) Appropriately grouped into subcategories (evaluated for potential breadth-and-depth exam) This process establishes the “defensible link” Must be approved by the EPE Committee 26

Examination Specification Based on Body of Knowledge 80 multiple-choice question Differentiates Software Engineers from other specialties Currently 26 different exams including Control Systems Electrical and Computer: Computer Engineering Electrical and Computer: Electrical and Electronics Electrical and Computer: Power 27

Examination Specification Electrical and Computer: Computer Engineering Computer Systems (40)% Numeric and Nonnumeric Formats Computer Architecture Hardware (25%) Digital Devices Digital Electronics Digital Circuits Hardwar Description Languages Software (25%) System Software Development/Applications Software Maintenance Networks (15%) Computer Networks Physical Layers Implementation 28

Examination Specification Electrical and Computer: Computer Engineering Software System Software Operating systems Real-time operating systems 3Computer security Device drivers Interrupts Development/Applications Software design and documentation methods Quality assurance Fundamental constructs Programming language characteristics Development tools Software Maintenance Configuration management Software update Change control 29

Examination Specification Software Engineering Requirements (17%) Design (14%) Construction (11%) Testing (13%) Maintenance (7%) Configuration Management (8%) Engineering Processes (7%) Quality Assurance (8%) Safety, Security and Privacy (15%) 30

Examination Specification Software Engineering Requirements Software requirements fundamentals (e.g., concept of operations, types of requirements, product and process requirements, functional and nonfunctional requirements, quantifiable requirements, system requirements, software requirements, derived requirements, constraints, service level) Requirements elicitation Requirements specification Requirements analysis Requirements verification and validation Requirements management 31

Examination Specification Software Engineering Engineering Processes Process definition (e.g., software life cycle models [Agile, Spiral, Waterfall, etc.], software life cycle processes, process tailoring, process assessment, process improvement) Software engineering standards (e.g., process, coding, development, testing, reliability, architecture) Estimation (e.g., software size and complexity, effort) Measurement (e.g., metrics definition, collection, and analysis, goal question metric) Risk management (e.g., addressing uncertainty, opportunities, decisions under risk, decisions under uncertainty) 32

Examination Specification Software Engineering Requirements (17%) Design (14%) Construction (11%) Testing (13%) Maintenance (7%) Configuration Management (8%) Engineering Processes (7%) Quality Assurance (8%) Safety, Security and Privacy (15%) 33

Examination Development Process Overview Item Writing & Review Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed 34

Item Writing and Review On-going process Sources of item writers Training Item submittal form Proper documentation Subject matter expert (SME) review Appropriateness of content Time to solve Solution and key Rationale for distracters 35

Examination Development Process Overview Examination Assembly & Review Item Writing & Review Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed 36

Examination Assembly & Review Item bank In accordance with the specification SME and committee review Content overlap Adherence to specification Consistency and sensitivity Timed pre-test 37

Examination Development Process Overview Standard Setting Study Item Writing & Review Examination Assembly & Review Examination Scoring Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed 38

Standard Setting Study The passing score is determined in one of two ways. For the first administration of a new exam or specification change, a cut score panel recommends the score. This becomes the “anchor” exam. For subsequent administrations, a statistical procedure known as “equating” is used to set the score relative to the anchor exam. 39

Examination Development Process Overview Equating of Examination Examination Administration Item Writing & Review Examination Assembly & Review Examination Scoring Standard Setting Study Need for Examination Identified Task Analysis Performed (PAKS) Examination Specification Developed After Cut Score Established 40

Current Status Licensure of software engineers now available in 43 states and jurisdictions Software P&P exam offered last 3 years (2013, 2014, 2015) Limited number of takers so far (12,14,16, respectively) Pass Rates are reasonable (63% for first time takers in 2015) Main issues are: Exam preparation (size of exam bank, etc.) Understanding when SW license is needed vs other specialties Spreading the word 41