Agile Development & Companies Harvard Business Review: “Why the lean start-up changes everything” Agile development applied to entire business of start-ups Simplest possible prototype! Iterate! Image: leadnet.org

Case Study: Case Study #1

Background My PhD thesis: new CAD System for FPGAs Formed company to commercialize 4 people initially An early customer was Altera 10 months to produce a new “placement and routing” system for Altera’s chips Aggressive goal: 10X better than current system

Place and Route Example

Project Management Quick plan Concrete milestones Scared 1 day: estimate all tasks & time Add time margin (mine: 50%) Concrete milestones Meet a milestone every 3 months or don’t get paid! Scared Turned out to be helpful! Avoid any side projects, unnecessary complexity Get something working, measure & improve Measure our software vs. Altera’s every day

Outcome Hit all milestones, except first (2 weeks late) Focused! CAD system exceeded expectations 30X less runtime 38% faster circuits Altera replaced their P & R engine with the prototype Started simple, measured where to improve Some simple algorithms still in current Quartus II  didn’t need more!

Case Study 2: (First Version of) Quartus

Background Altera had highly successful CAD system: Max+PlusII Decided to do a complete re-write to a new CAD system Quartus Goals: C++ (not C) Cleaner, more extensible code Allow multiple engineers to collaborate on a project easily Allow fast, “incremental” recompiles

Project Management Planned for a year with no coding Too much waterfall  planning paralysis Let complexity reign Fancy user features Fancy language (object oriented C++ to the max) No working prototype for ~2 years Can’t test new features in full system Didn’t understand & measure key metric CAD system that optimizes well And is ready for next-generation chip

Outcome Software not ready in time for chip Software rushed to market Not stable enough, didn’t optimize well enough Very bad customer experiences Lost sales: $billions! Rewrote & renamed later versions: Now very good!

Case Study 3: Hardware + Software (Stratix V)

State of the art for 2 years! Big Project! More planning? 3 year project ~400 engineers at peak Yes, plan as far ahead as you can But don’t paralyze yourself Accept that plan gets coarse as you go out in time Key plan: measurable milestones 3 year project  need to break up schedule Project milestones  goals / milestones per team Clear must have features Everything else: nice to have > $200 M State of the art for 2 years! +

Agile / Iterative Development? Build prototype / model of HW + SW system Use to evaluate and test new ideas Done by smaller team (~10 engineers) Will never ship to customers! Prototype shows good results? Build real hardware & full software Team grows to 400 Always keep system working as you improve Always measure results vs. expected (from prototype) HW & SW both in source code control Still iterative, but with more structure

Communication? Weekly status meeting Meeting style For sub-teams (~8 engineers) Managers discuss results for ~8 teams (~64 people) Directors meet to discuss overall project status Meeting style wiki, crisp reporting Big picture  progress toward milestones

Outcome Iterative development of HW + SW Works much better than pure waterfall Many companies: out of business by not getting full HW + SW system working Every 2nd generation: usually late! Engineers & marketing put in too much complexity Not scared enough, not enough iterative development! Late to market is terrible Competitor has twice the transistors, 1 year earlier Want to optimize, but on time more important

Testing

Testing Philosophy “If you didn’t test it, it doesn’t work” Assume all code broken until proven otherwise Look at what the program does not what it is supposed to do Scientist: you are testing hypotheses about program Think of every case your program should handle  cover them all

Testing Time Develop tests as or before you code Test-driven development Prototype Refine Test & Evaluate

Test-Driven Development The tests are the detailed specification And now can be executed, not just read More tests Fewer specification documents

Types of Tests End-to-end (system) tests Tests like an end user Hard to debug a failure  problem could be anywhere! Fragile: often test some exact user input would produce some exact output text / file End to End Tests Unit Tests Test small pieces of system (single classes / functions) Easier to debug failure  start with 1 class / function Don’t check user interface (I/O): check API / class does what it should with code (testfixture) Unit Tests

Types of Tests Fewer Integration Tests Bigger unit tests (but not 0!) End to End Tests Integration Tests Bigger unit tests But now test multiple classes / bigger functionality Moderate difficulty to debug Integration Tests Unit Tests More

Testing Automation You will re-test often System tests: save in files Every time you change the program How to speed up? System tests: save in files Use file redirection prog.exe < in.txt diff out.txt out_good.txt // matched known good result? Less fragile Write validity checker instead of checking exact output match Write scripts to run all tests and check results > out.txt

Unit Tests Want: No! To test individual classes / functions How? Carefully chosen user input? Output ... No! struct Point { float x; float y; t_point& operator*=(float rhs); . . .

Unit Tests Write new code To put class in right state To directly send it some input / make some calls To immediately check the responses Test driver

Unit Tests myCode.cpp struct Point { float x; float y; t_point& operator*=(float rhs); . . . tester.cpp int test1 () { Point testme (1, 2); testme *= 3; if (testme != Point (3, 6)) { cout << “uh – oh” << endl; return (1); } return (0); test_main.cpp g++ myCode.cpp main.cpp –o prog.exe int main () { int error = 0; error += test1(); . . . // Lots more tests g++ myCode.cpp tester.cpp test_main.cpp –o test.exe