1. OPEN DEVELOPMENT, Agile, xp AND SCRUM
COMP 319
© University of Liverpool
COMP319

2. © University of Liverpool
Linux a brief history
GNU project
Richard Stallman
C compiler released in 1987
Minux
Andrew S. Tanenbaum
12,000 lines of C and 8086 assembler
Linux
Linus Torvald 1991
Linux distribution today
Linux kernel (3%) + GNU software (28%) + others
Minux is an example OS written by Andrew Tanenbaum, and was provided to purchasers of his book, 'Operating Systems: Design and Implementation‘.
Quote from Linus Torvald (posting to Minux newsgroup, 25th August 1991)
“Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones…”
Cost estimations for Linux have put its total development costs if done commercially at over 1 billion USD (this was in 2001),
and estimations for its development time to be over 8000 person/years.
Note also a large percentage of any Linux distribution including the compiler and some of the GUI contributions have come from
the GNU project. For example Linux kernel % of a typical distribution is around 1.5 to 3%.
In fact distributions typically have software from many different sources.
COMP319
© University of Liverpool
COMP319

3. Cathedral and the Bazaar
Eric Raymond in 1997 published The Cathedral and the Bazaar
He conjectured that there were essentially two ways to engineer software
1. In the cathedral, and
2. In the bazaar
In Eric Raymond’s (1997) paper he notes the emergence of ‘bazaar’ based software engineering. The paper is essentially his observations on the use of that approach to develop the ‘fetchmail’ system.
Up to that time he had thought there there was just complex corporate systems and more private, ‘hacked’, systems – constructed by lone gurus (hackers, in the sense of hacking a solution, rather than hacking at a system).
He had thought that once a system (such as an operating system) went beyond some level of complexity, then it could only be constructed like a cathedral, by a team of cathedral builders.
(In the original article the impression was more that what was produced was written in the hallowed confines of a cathedral by a team of high priests … but he thought that too derogatory … and a new version emerged, that had a new spin on who the builders were …)
COMP319
© University of Liverpool
COMP319

4. Cathedral approach (e.g. IBM, MS)
Leads to large complex programs such as operating systems
These projects were worked on by teams of “high-priests/cathedral builders”
The product was essentially completed when released for sale
Were the province of large organisations
Raymond’s view was that the cathedral approach, happened after a certain critical complexity level was reached.
Above this level, it was necessary to have a team of high priests/cathedral builders who worked on a system until it could be released in all its glory.
He believed that this level of complexity was typical of modern operating systems and other complex systems produced in the commercial world.
The need to sell a product or having some significant customer (military, government, another company) meant that until the release only those in the company working on the system got to see it.
COMP319
© University of Liverpool
COMP319

5. © University of Liverpool
The Bazaar Approach
A large network of communicating developers all interested in the solution
“Release early. Release often. And listen to your customers” - Linus’ principle
“Given enough eyeballs, all bugs are shallow'' - Linus' law coined by Raymond
Source freely available and everyone encourage to take part as equals
Raymond observed that with the development of Linux the idea that complex software could be constructed only in (or like) a cathedral was wrong.
He noted that coinciding with the development of the internet were groups of open source developers working together on a solution to common problems, and that sometimes these problems were highly complex systems.
What seemed to happen was that some central architect emerged (like Linus Torvalds – Linux) and put together some attractive initial solution that worked and which was quickly released. This was used and corrected, then rebuilt and rebuild was released quickly. The idea was enshrined in Torvald’s principle <read>
The problem and the solutions were shared; poor solutions were pruned by peer review and the central architect.
Frequent releases meant that thousands of user eyes were looking for bugs, rather than just the developers eyes. This was the law behind Torvald’s principle – which Raymond called Linus’ law <read>
Although the releases were coordinated by Torvalds, everyone was enabled to contribute. This was achieved by making the source fully and freely available
COMP319
© University of Liverpool
COMP319

6. Open source development
All code open to peer review
Large tester base (need to early and frequent code releases, early Linux was released daily)
Open source code/testing debugging
testing frameworks
Run the code using source code debugger
Testing in an open source environment provides an extra level of depth.
Testers can run the code using debugging/tools and logging.
If code is compiled using debugging options when exceptions are thrown the source line causing the problem can be identified.
COMP319
© University of Liverpool
COMP319

7. Case study (fetchmail)
Raymond set out to discover why the bazaar approach worked
About increasing personal productivity
About open source, open management, open goals
“egoless programming” Weinberg
Raymond discovered that the approach did work. Rapid development of complex software such as his POP3 client (fetchmail) was possible.
He discovered that his personal productivity was increased by adopting an open source, open management, and open goals approach. This was because he shared problems with his community and adopted solutions and approaches happily and willingly.
His observation was that personal software productivity, and the development of new software would benefit from adopting the basic rules of the bazaar approach.
He noted that what he was proposing was not particularly new. Weinberg, G.M. (1971) in The Psychology Of Computer Programming (Van Nostrand Reinhol), had suggested the idea of “egoless programming”, the idea that software engineering was a shared process. The idea had been dismissed because of the choice of the word “egoless” – Raymond considered it unfortunate and possibly inaccurate, since he noted it was sometimes because there were significant egos (that needed and liked being stroked by equals) that progress was made. His comment was “…who likes to be told it cannot be done, or cannot be done in 25 lines …”.
COMP319
© University of Liverpool
COMP319

8. © University of Liverpool
fetchmail Lessons 1-6
Personal commitment to the software
Use what is available
One will get thrown away
The individual (interest) is central
Interest is not necessarily sustained
Users like helping; become developers
Raymond then listed the 19 lessons, sometimes restating them. However, it is worth listing them all and better to read his paper for an explanation of why they arise.
The main one, is without doubt commitment to the ideas behind the software. In his case an efficient mail system – which did not properly exist at the time.
He looked around for what already existed, modified it with a new architecture, chatted to the original designer who told him what he had intended putting in it, modified it again, then released it. It was liked and a group of committed and interested people gathered.
Very early on he planned to rewrite the system, so “throw one away” a phrase from MMM, was already in his plans.
Interest, see commitment. But note, it is is difficult to sustain interest. You must be prepared to hand things off to others to continue.
Lesson 6 is very important. Users do like being involved, and if they have access to the source and know about programming they will make useful suggestions regarding bugs and fixes, as well a features.
COMP319
© University of Liverpool
COMP319

9. © University of Liverpool
fetchmail Lessons 7-11
Release early & often. Listen to your customers
Many eyes make debugging light
Get data structures right, code follows
Testers are your most valuable resource
Recognise good ideas from others
COMP319
© University of Liverpool

10. © University of Liverpool
fetchmail Lessons 7-11
Release early & often. Listen to your customers
Many eyes make debugging light
Get data structures right, code follows
Testers are your most valuable resource
Recognise good ideas from others
Lesson 7 is the golden Linus rule. Release as soon as you have something that works and might be of interest … if it is, then listen to the users/customers. Lesson 8, follows and is Linus’ law (which you will note can be parsed in many ways …).
Lesson 9 has been noted by every software commentator and engineer and bears being restated many many times.
Lesson 10 is again a restatement, but it brings to the fore the value of “many eyes”.
Lesson 11 is the “egoless” point from the central architects point of view.
COMP319
© University of Liverpool
COMP319

11. © University of Liverpool
fetchmail Lessons 12-17
Recognise your poor ideas
Design perfection comes from pruning
Good software is used in unexpected ways
Don’t throw information away
Beware of pseudo-secrets
And lesson 12 is a restatement, again the egoless point, but rephrased to emphasise that one ego (the architects) may get in the way of good ideas and rapid development.
Lesson 13 simply restated is that “good design is simple design”. Good design is also “beautiful” as Brooks noted. Both, simplicity and beauty, come from pruning – as any rose grower will tell you.
When your spreadsheet is used as a database, you’ve got a good spreadsheet package.
Lesson 15 is based on technicalities of fetchmail, but the core idea is that if some software acquires information, it is wise not to discard it just because it has no immediate purpose. For example if 8 bits of information are always sent to you and 4 are rarely used, don’t use the 4 bits for your purposes now, wait, it may be that someone will want a feature that depends on the occasional rare use of them. Put another way, always keep an untouched version of your input data, you may need it.
Lesson 16 is about the extra statements that are added to a language to make it easier to read or write a program, effective use of these constructs is important when producing easy to read programs. Example foreach.
Lesson 17 is about security. When you need it, think about it and do it carefully, do not add false security (e.g. usernames and passwords) just for the sake of it.
COMP319
© University of Liverpool
COMP319

12. Syntactic sugar is your friend
public int X{ // C#
get { return x; }
set { x = value; } }
Generics (always use if relevant!)
For each Java
ArrayList <Person> list=new ArrayList <Person>();
for (Person person : list) {
Make’s code tighter
COMP319
© University of Liverpool

13. © University of Liverpool
Surgical Teams
Observations on team working – Brooks
Roles: surgeon, co-pilot, administrator, editor, 2 secretaries, program clerk, toolsmith, tester, language lawyer.
10 roles; around two surgeons one taking the lead and dedicated to the project
Teams of 10 do the surgery
This is the perspective from 1972, having many co-workers to support a specialist development, currently this list
Looks heavily bureaucratic, but programming at this time was long winded and difficult, there were no proper
Development environments.
Nowadays in fact many of the roles in this list are now done using some degree of automation for example supported by an IDE.
Brooks here is all about mixing metaphors … and is intended to be a description of recommended best practice in the cathedral approach. Where, in contrast to the bazaar approach, software (maybe novel) but certainly large in size, must be generated to schedule for good commercial reasons.
Here everyone is paid because there is a profit generated and motivation is deemed to be the salary, an interesting job, and a worthwhile product which is used and successful.
Within the team all property and responsibility is shared and this ownership extends to success or otherwise – which generates motivation.
Of the 10, 7 are professionals, expert in their own areas. The aim with teams of this kind is to minimise communication between members we will return to this point. First off we examine the roles <click>
COMP319
© University of Liverpool
COMP319

14. © University of Liverpool
1970s Perspective
PDP 11
Max 128K RAM
Cost about £12,000 then, equivalent of around £124,000
Applications at the time
Therac-25 radio therapy
Air traffic control
We can see that very modest hardware in the 1970s was used to perform highly complex tasks.
To achieve this the degree of work which would be required to:
Make the system perform fast enough.
Fit the software into limited memory.
Debug and test the software even though the tools available were very limited.
COMP319
© University of Liverpool
COMP319

15. © University of Liverpool
The Surgeon role
Involved in overall system architecture
Defines functional and performance specification
Writes documentation
Uses HLL and CASE tools
Experienced and well paid
The surgeon (chief programmer, lead programmer, …) personally defines the functional and performance specification then designs and writes the code and first draft of the documentation.
She is captain of the project and makes all the final decisions although they may executed by others (e.g. administrator or editor).
She should be highly experienced, well paid, and considered a senior person in the company.
COMP319
© University of Liverpool
COMP319

16. © University of Liverpool
Co-Pilot Role
Can do whatever the surgeon can
Shares in the design as a thinker, discussant, and evaluator
Represents the team and acts as interface
May write code but is not responsible for it
Less experienced, younger
The co-pilot oversees what is being done and is there to discuss and act as an evaluator of what is being implemented.
He researches alternative approaches and designs, and acts as the sounding board for ideas suggested by the surgeon.
He is also there in case the surgeon falls under a bus.
Paid less and is training to be a surgeon.
COMP319
© University of Liverpool
COMP319

17. Administrator, editor, support
Administrator deals with staff, money, space. Liaises with the rest of the organisation. Usually serves two teams.
Editor generates final documentation, nurses it through to publication
Clerical and secretarial support is vital for the administrator and editor
This role is not so obvious in
These roles are not necessarily associated with just one team (except perhaps the two secretaries).
In the case of the administrator he may have two teams to deal with.
In the case of the editor, his involvement may begin after the team has started.
Again from a modern perspective we can see some of these functions are expected to be done by the
programmer with help of the tools available.
COMP319
© University of Liverpool
COMP319

18. Program Clerk, Toolsmith
Clerk maintains all technical records as a librarian and as a secretary is responsible for machine, and paper files
Toolsmith is an expert who evaluates, upgrades, customises, builds tools as required by the surgeon and team
The program clerk logs and maintains all files and ensures that these data are available to any member of the team. He maintains the shared data of the team as a whole.
The toolsmith will be an expert dedicated to the team although he may work for several teams at the same time.
COMP319
© University of Liverpool
COMP319

19. Tester, Language Lawyer
Tester generates test cases and writes the test environment. S/he confirms that the functional requirements are met
Language Lawyer is the HLL expert who thinks up neat, efficient ways to do difficult and obscure things. Will be constantly researching good technique. One lawyer serves several teams
The tester is the person who checks that the functional requirements of the system are met.
He builds test environments and specifies test cases and data to test the various stages of the development.
The language lawyer helps by researching specific algorithms required by the team and keeping abreast of ways of improving beauty, simplicity, and efficiency of generated HLL code. May specialise in data structures and/or methodological approaches – e.g. OO. Language lawyers are usually shared by several teams.
COMP319
© University of Liverpool
COMP319

20. Surgical team structure C21st
Software team leader and junior programmer
Tester
Project manager/administrator
Everything else done by
IDE and debugging, re-factoring tools
OO patterns
Source control systems
Structure has changed in the 21st century but still the idea of the senior developer is very useful.
Nowadays the move towards more robust HLLs and tools has allowed detailed technical skills to be
replaced by automation.
COMP319
© University of Liverpool
COMP319