Can We Estimate Software Development Time? Chuck Connell Copyright 2002, Charles H. Connell Jr.

Background This talk is a response to a paper by J.P. Lewis titled Large Limits to Software Estimation in ACM Software Engineering Notes, July This talk is a response to a paper by J.P. Lewis titled Large Limits to Software Estimation in ACM Software Engineering Notes, July 2001.Large Limits to Software EstimationLarge Limits to Software Estimation Lewis’s paper generated some good discussion within the software engineering community and on Slashdot.org. Lewis’s paper generated some good discussion within the software engineering community and on Slashdot.org.Slashdot.org My article on this topic appeared on Developer.com and Slashdot in January My article on this topic appeared on Developer.com and Slashdot in January 2002.article

Outline Motivation for the topic Motivation for the topic Summary of Lewis’s paper Summary of Lewis’s paper My critique of Lewis’s paper My critique of Lewis’s paper

Why estimating matters Estimating the size of future projects is one of the most important areas of software engineering. (Size = $$, person-days, calendar days, opportunity cost.) Estimating the size of future projects is one of the most important areas of software engineering. (Size = $$, person-days, calendar days, opportunity cost.) Estimates are crucial to go/no-go decisions on every project. (Example: $1M project for this group.) Estimates are crucial to go/no-go decisions on every project. (Example: $1M project for this group.) Questions about the ability of SE to provide meaningful estimates call into question all plans for future projects. (And all current projects.) Questions about the ability of SE to provide meaningful estimates call into question all plans for future projects. (And all current projects.)

SE status within CS Software engineering’s ability to estimate is part of a broader discussion about the overall validity of SE. Software engineering’s ability to estimate is part of a broader discussion about the overall validity of SE. If SE cannot make good estimates about future projects, what is the whole field doing? If SE cannot make good estimates about future projects, what is the whole field doing? SE is frequently derided by computer scientists as unscientific, non-rigorous, or “just programmer sociology”. SE is frequently derided by computer scientists as unscientific, non-rigorous, or “just programmer sociology”. I address this broad question in Why Software Engineering Is Not B.S., inspired by a CS professor’s remark. I address this broad question in Why Software Engineering Is Not B.S., inspired by a CS professor’s remark.Why Software Engineering Is Not B.S.Why Software Engineering Is Not B.S.

Lewis: what software is The behavior of any program can be thought of as a table of inputs and outputs. The behavior of any program can be thought of as a table of inputs and outputs. There is one row for each possible set of inputs. (Example: 0-3 x 0-3) There is one row for each possible set of inputs. (Example: 0-3 x 0-3) Assumes finite number of finite inputs. Assumes finite number of finite inputs. Assumes we can represent human interaction as another finite/finite input. Assumes we can represent human interaction as another finite/finite input. Assumes finite/finite output for each set of inputs. Assumes finite/finite output for each set of inputs. Above are reasonable assumptions for practical purposes. Above are reasonable assumptions for practical purposes.

Lewis: what software is Real software would have a very large table. Real software would have a very large table. Something like: Something like:  2 32 raised to # of numeric inputs, times  2 8 raised to # of text input chars But the table is finite and mechanically constructible. But the table is finite and mechanically constructible.

Lewis: software spec as a string A software description is a finite table. A software description is a finite table. Any such table can be transformed into a string. Any such table can be transformed into a string. The string represents what we want the software to do. The string represents what we want the software to do. Example: using previous Example: using previous So a software specification is a finite string. So a software specification is a finite string.

Lewis: estimating software tasks Estimating a software task is the same as asking, “How hard is it to write a program to create the spec string?” Estimating a software task is the same as asking, “How hard is it to write a program to create the spec string?” (We assume a small driver to call program with every input.) (We assume a small driver to call program with every input.) Lewis claims that writing a smaller program is easier/faster than writing a larger program. Lewis claims that writing a smaller program is easier/faster than writing a larger program. So, Lewis asserts that estimating is equivalent to asking: “What is the smallest program (in bytes) to create spec string?” So, Lewis asserts that estimating is equivalent to asking: “What is the smallest program (in bytes) to create spec string?”

Lewis: complexity of estimating What is the smallest program to create spec string? What is the smallest program to create spec string? By definition, this is identical to asking for the algorithmic complexity of the string (related to Kolmogorov complexity). By definition, this is identical to asking for the algorithmic complexity of the string (related to Kolmogorov complexity). algorithmic complexity algorithmic complexity But, there is a well-known result from complexity theory: There is no program to compute the algorithmic complexity of arbitrary strings. But, there is a well-known result from complexity theory: There is no program to compute the algorithmic complexity of arbitrary strings.complexity theorycomplexity theory Therefore, there is no rigorous method for estimating software development! Therefore, there is no rigorous method for estimating software development! Lewis extends this result to estimates with a given percentage accuracy. Lewis extends this result to estimates with a given percentage accuracy.

Summary of Lewis’s paper Software specs are tables / strings. Software specs are tables / strings. It is faster to write small programs than large programs. It is faster to write small programs than large programs. Software estimates seek the algorithmic complexity of the specification string. (What is the smallest program to print out the spec table?) Software estimates seek the algorithmic complexity of the specification string. (What is the smallest program to print out the spec table?) But there is no method for finding the algorithmic complexity of arbitrary strings. But there is no method for finding the algorithmic complexity of arbitrary strings. Therefore, rigorous/mechanical software estimates are not possible, even allowing % error. Therefore, rigorous/mechanical software estimates are not possible, even allowing % error.

My critique Lewis is “correct”, given the definitions he uses and the technical results he obtains, but… Lewis is “correct”, given the definitions he uses and the technical results he obtains, but… He mischaracterizes the programming process. He mischaracterizes the programming process. He fails to show a relationship between algorithmic complexity and software development times. He fails to show a relationship between algorithmic complexity and software development times. He mischaracterizes promises made by well- known development frameworks, especially the Software Capability Maturity Model (CMM). He mischaracterizes promises made by well- known development frameworks, especially the Software Capability Maturity Model (CMM).CMM

Programming process Do programmers try to write the shortest possible program? Do programmers try to write the shortest possible program? Maybe, sometimes. Maybe, sometimes. But modern software engineering teaches that other aspects of software are more important: readability, modifiability, scalability, speed… But modern software engineering teaches that other aspects of software are more important: readability, modifiability, scalability, speed… The shortest program may not have these qualities. The shortest program may not have these qualities. So programmers may not be trying to do something directly related to the AC of the spec table. So programmers may not be trying to do something directly related to the AC of the spec table.

Estimation problem Assume Lewis’s description of software algorithmic complexity (AC) is well-defined… Assume Lewis’s description of software algorithmic complexity (AC) is well-defined… Does it take longer for people to write a program for a high AC task than for a lower AC task? Does it take longer for people to write a program for a high AC task than for a lower AC task? The answer to the above must be YES, or AC is unrelated to the software estimation problem. The answer to the above must be YES, or AC is unrelated to the software estimation problem. But we don’t know the answer to this question, a priori. But we don’t know the answer to this question, a priori.

Estimation problem Does it take longer for people to write a program for a high AC task than for a lower AC task? Does it take longer for people to write a program for a high AC task than for a lower AC task? This is an empirical question for psychology. This is an empirical question for psychology. Maybe there are tricky low-AC tasks that are hard (long) to program… Maybe there are tricky low-AC tasks that are hard (long) to program… Maybe there are high AC tasks that a human can grasp quickly… Maybe there are high AC tasks that a human can grasp quickly… So programming estimates may not be directly related to differences in AC. So programming estimates may not be directly related to differences in AC.

Claims about estimating Lewis attacks a straw man: objective, mechanical estimating procedures with guaranteed accuracy. Lewis attacks a straw man: objective, mechanical estimating procedures with guaranteed accuracy. It is true that some writers (or book marketers) have appeared to offer such methods, but no serious researchers are trying to achieve this. It is true that some writers (or book marketers) have appeared to offer such methods, but no serious researchers are trying to achieve this. Software engineering researchers are really looking for an estimating method that is somewhat accurate, most of the time. Software engineering researchers are really looking for an estimating method that is somewhat accurate, most of the time.

Claims about estimating Imagine an estimating technique that is 80% right 80% of the time. Imagine an estimating technique that is 80% right 80% of the time. For the other 20%, the estimate is worthless. For the other 20%, the estimate is worthless. Everyone in the software business would be leaping with joy! Everyone in the software business would be leaping with joy! Such average-case algorithms are well-studied in computer science, and have been used for prime number tests and solving simultaneous equations. Such average-case algorithms are well-studied in computer science, and have been used for prime number tests and solving simultaneous equations.average-case algorithmsaverage-case algorithms Lewis’s mathematical arguments do not disallow such a solution, because of the unbounded errors. Lewis’s mathematical arguments do not disallow such a solution, because of the unbounded errors.

Claims about estimating In particular, Lewis claims that CMM promises guaranteed accuracy of software estimates, and provides a quote from CMM to show this. In particular, Lewis claims that CMM promises guaranteed accuracy of software estimates, and provides a quote from CMM to show this. In fact, CMM promises no such thing; the quote is taken out of context. In fact, CMM promises no such thing; the quote is taken out of context. CMM does predict better estimates on average the more closely one follow CMM guidelines. (A claim which may or may not be true.) CMM does predict better estimates on average the more closely one follow CMM guidelines. (A claim which may or may not be true.) Lewis is not alone in this error; because CMM is so dominant, it is often perceived to be more rigid than it is. Lewis is not alone in this error; because CMM is so dominant, it is often perceived to be more rigid than it is.

How should we estimate software projects? 1. Expert judgment based on experience with the project particulars. (Example from Lotus NLM.) 1. Expert judgment based on experience with the project particulars. (Example from Lotus NLM.) 2. Iterative development, with iterative estimates (e.g. study, POC, prototype, R1, R2, etc.) 2. Iterative development, with iterative estimates (e.g. study, POC, prototype, R1, R2, etc.) Iterative development accepts the fact that large, complex projects cannot be estimated up front. Therefore don’t do large, complex projects all at once. Iterative development accepts the fact that large, complex projects cannot be estimated up front. Therefore don’t do large, complex projects all at once. See Rapid Development by Steve McConnell. See Rapid Development by Steve McConnell.Rapid Development Rapid Development

Questions ?