Probabilistic project sizing using Randomized Branch Sampling (RBS) Dimitar Bakardzhiev Managing Director Taller Technologies Bulgaria @dimiterbak

Dimitar Bakardzhiev is the Managing Director of Taller Technologies Bulgaria and an expert in driving successful and cost-effective technology development. As a Lean-Kanban University (LKU)-Accredited Kanban Trainer (AKT) and avid, expert Kanban practitioner, Dimitar puts lean principles to work every day when managing complex software projects with a special focus on building innovative, powerful mobile CRM solutions. Dimitar has been one of the leading proponents and evangelists of Kanban in his native Bulgaria and has published David Anderson’s Kanban book as well as books by Eli Goldratt and W. Edwards Deming in the local language. Thank you very much for your attention. My hope is that you will start using this approach for high-level planning your next project. Dimitar Bakardzhiev is the Managing Director of Taller Technologies Bulgaria and an expert in driving successful and cost-effective technology development. As a Lean-Kanban University (LKU)-Accredited Kanban Trainer (AKT) and avid, expert Kanban practitioner, Dimitar puts lean principles to work every day when managing complex software projects with a special focus on building innovative, powerful mobile CRM solutions. Dimitar has been one of the leading proponents and evangelists of Kanban in his native Bulgaria and has published David Anderson’s Kanban book as well as books by Eli Goldratt and W. Edwards Deming in the local language. He is also a lecturer and frequent speaker at numerous conferences and his passion is to educate audiences on the benefits of lean principles and agile methodologies for software development. @dimiterbak

How big is our project? In order to forecast the time and the budget needed to deliver a new software product we need to be able to quantify “what” we are building since the resources required are related to “how much” software is built. That quantification is referred to as “sizing”.

Project sizing is different from delivery time estimation Project sizing is different from delivery time estimation. Sizing estimates the probable size of a piece of software while delivery time estimation forecasts the time needed to build it. The relationship between the size of a piece of software and the time needed to deliver it is referred to as productivity. Software sizing is different from delivery time estimation. Sizing estimates the probable size of a piece of software while delivery time estimation forecasts the time needed to build it. The relationship between the size of a piece of software and the time needed to deliver it is referred to as productivity.

Agile sizing techniques measure User Stories T-Shirt sizes (Small, Medium, Large and so on) First we have to decide in what units to measure project size. In the past the “amount of product” produced from a software development project was perceived as being the number of source lines of code written. Later on Function point analysis (FPA), which measures the size of the software deliverable from a user’s perspective, was introduced. FPA even made it to be codified in several ISO standards. However FPA is not used in Agile projects where T-Shirt sizing and story points are the favorite ways to estimate how big a user story is. The primary advantage of using t-shirts sizes is the ease of becoming accustomed to sizing. Their primary disadvantage is that they are not additive. We cannot say the size of a product is 3 mediums (M), 4 larges (L) and two smalls (S) (Cohn, Estimating with Tee Shirt Sizes, 2013). http://www.mountaingoatsoftware.com/blog/estimating-with-tee-shirt-sizes Story points (Fibonacci numbers or an Exponential scale) http://www.mountaingoatsoftware.com/blog/dont-equate-story-points-to-hours

Story points are about effort (Cohn). But…software sizing is different from software effort estimation! http://www.mountaingoatsoftware.com/blog/story-points-are-still-about-effort

Sizing estimates the probable size of a piece of software while effort estimation estimates the effort needed to build it.

Kanban project sizing? Count! Number of user stories, features, use cases Number of tasks Any type of work items suitable for the development organization.

Example of project sizing in tasks We have identified 16 epics in our project We have identified that those 16 epics contain 102 user stories in total We have analyzed and sized every single one of those 102 user stories and arrived at a total number of 396 tasks for our project To plan delivery of initiatives, features or epics we have to break them down into user stories. That requires quite significant analysis process and it can easily happen that great part of this effort will be a pure waste. Priorities of initiatives and features can change and feature could not be started at all. This practice is hard to follow. This practice is time consuming and probably great part of this effort will be a pure waste!

How can we estimate the total number of stories or tasks for a project without prior identification, analysis and sizing of every single user story? It turns out there is such a technique called Randomized Branch Sampling (RBS) first proposed by Jessen in 1955.

Randomized Branch Sampling RBS is a multi-stage unequal probability sampling method. The technique was designed to efficiently estimate the total number of fruit found in the canopy of a tree while only having to count the fruit on select branches. http://senate.universityofcalifornia.edu/inmemoriam/RaymondJessen.htm Randomized branch sampling (RBS) was first proposed by Jessen (Jessen, 1955). It is a multi-stage unequal probability sampling method. The technique was designed to efficiently estimate the total number of fruit (oranges) found in the canopy of a tree while only having to count the fruit on select branches. RBS is a method for sampling tree branches which does not require prior identification of all branches, and provides the sampler with unbiased tree level estimates. As an example, we suppose that it is of interest to estimate the total number of fruit on a apple tree. The number of fruit on the tree is the population and it is desired to select a sample from which an estimate of the population's number can be obtained. In the case of the apple tree, the samples to be selected are the terminal branches upon which the fruit is typically borne. Raymond J. Jessen 1910–2003

Product backlog as a branching system With RBS, branches are selected from the tree by creating a pathway which starts at the base of the trunk and travels upwards. In order to apply RBS for sizing a software project we will represent a product backlog as a branching system as follows: The product backlog is the work that needs to be accomplished to deliver a product with specified capabilities (features and functions). The features and functions are called requirements and are presented and managed using vehicles such as epics and user stories. Epics are the highest-level requirements artifact. Epics are not implemented directly but are broken into user stories, which are the primitives used by the teams for actual coding and testing. ‹‹ Epics are not directly testable. Instead, they are tested by the acceptance tests associated with the user stories that implement them. When breaking an epic down do not consider how many steps the workflow will consist of. Consider the user story the only work item type at the lowest level of granularity. Epic A Epic B Epic C User Story 1 User Story 2 User Story 5 User Story 6 User Story 7 User Story 3 User Story 4

𝑋 = 𝑥 𝑖 𝑄 𝑖 Horvitz-Thompson estimator 𝑋 = 𝑥 𝑖 𝑄 𝑖 In RBS a Horvitz-Thompson estimator is used to derive an unbiased estimate of the total size of the product backlog by dividing the size (x) of a selected user story i by its associated unconditional probability ( 𝑄 𝑖 ) with which that particular user story was selected. Unbiasedness generally means that the average of the estimates over all possible samples will be identical to the actual project size. An unconditional selection probability ( 𝑄 𝑖 ) is obtained from the conditional selection probability of the 𝑖 𝑡ℎ user story when the conditional probabilities of the product backlog and the epic the story belongs to are accounted for. Since there is only one product backlog its conditional selection probability is 1. Unconditional selection probability ( 𝑄 𝑖 ) 𝑄 𝑖 = 𝑘=1 𝑖 𝑞 𝑘 = 𝑞 𝑝𝑟𝑜𝑑𝑢𝑐𝑡 𝑏𝑎𝑐𝑘𝑙𝑜𝑔 𝑞 𝑒𝑝𝑖𝑐 𝑞 𝑠𝑡𝑜𝑟𝑦 = 𝑞 𝑒𝑝𝑖𝑐 𝑞 𝑠𝑡𝑜𝑟𝑦

Total size per section level Application of RBS to project sizing Product Backlog Epic User Story Total size per section level The user story rectangle represents the estimated size of a randomly sampled user story. The size of that user story is expanded to an estimated total project size by dividing that size by its selection probabilities which is indicated here by the arrows.

Is RBS applicable to software development? The assumption behind using RBS for software development is that project size depends on the context – the customer, the people developing the product and the methodology they use for managing the requirements. It doesn’t matter what the methodology is. What is important is the methodology to be cohesive, explicit and to be consistently applied during project execution when we slice the requirements into user stories. The number of fruit found in the canopy of a tree has an inherent, abstracted from its interpretation, objective value. Counting all the fruit should always produce the same number plus or minus some counting error. When we apply RBS to find the number of fruit the fruit is already there. The size of a software project has no objective value. It is intangible - a proxy for all the capabilities (features and functions) the final product is required to offer when delivered to the customer. But requirements on a software project change over time and for a good reason. We are learning as we work on the project. We will discover new requirements and decide others are no longer valid (Anderson, 2011). Hence at the beginning of a project when we apply RBS to forecast its size the user stories are not available yet. The full set of user stories will be there when we finish the project. That leads to the question – from what universe of user stories we sample using RBS? The assumption behind using RBS for software development is that project size depends on the context – the customer, the people developing the product and the methodology they use for managing the requirements, breaking down the product into stories and sizing a story. It doesn’t matter what the methodology is – Planning Poker (Cohn, 2005), Product Sashimi (Rainsberger, 2012), Behavior Driven Development (North, 2006), Feature Driven Development (Coad, 1999) etc. What is important is the methodology to be cohesive, explicit and to be consistently applied during project execution when we slice the requirements into user stories. RBS estimates the original scope. Then there is a relationship between the original and eventual (emergent, expanded) scope which we only identify when the results of the execution of our plans meet the customer). The scope expansion occurs when the requirement change risk materializes.

The method of analysis used to break down requirements and itemize them for development has its own degree of variability. DJA, Kanban, p.221

RBS compared to the actual results of 13 real ScrumDo.com projects Epic-Story-Task breakdowns Successful release history Stable teams (systems) Have an active ScrumDo coach or scrum master Commercial projects Have a minimum size of 12 epics/features. I teamed up with Ajay Reddy and the CodeGenesys/ScrumDo.com team to test the correlation between project sizes estimated using RBS and the actual story points estimated in thirteen randomly selected projects from the pool of real Scrumdo projects that met the following criteria: Epic-Story-Task breakdowns Successful release history Stable teams (systems) Have an active ScrumDo coach or scrum master Commercial projects Have a minimum size of 12 epics/features. As seen on the scatterplot below we found a very strong correlation between project sizes estimated using RBS and the actual number of stories estimated for some real Scrumdo projects.

RBS estimated number of stories As seen on the scatterplot below we found a very strong correlation between project sizes estimated using RBS and the actual number of user stories for the real Scrumdo projects.

RBS estimated total story points ScrumDo data and results here.

RBS estimated number of tasks per story

Conclusions from Scrumdo.com data During project execution all project teams consistently applied a methodology for slicing the requirements into user stories and tasks During project execution all project teams maturely managed the emergent and high-change-risk requirements Execution is more important than planning!!!

Applications of RBS Applying RBS for estimating total number of user stories in a project Applying RBS for estimating total number of tasks in a project Applying RBS for estimating total number of BDD scenarios in a project Assessing the maturity of teams in their usage of a sizing methodology

Applying RBS for estimating total number of user stories in a project

Stories based sizing model Product … Epic 1 Epic 2 … User Story 1 User Story N

… User Story 1 User Story N … Epic 1 Epic N Product

Mapping Product Trunk Epic Branch User Story Terminal Shoot Mapping

RBS estimate of the of total number of user stories for a project 𝑋 𝑖 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑢𝑠𝑒𝑟 𝑠𝑡𝑜𝑟𝑖𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝐸𝑝𝑖𝑐 1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐸𝑝𝑖𝑐𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡 (1) Where: 𝑋 𝑖 is an estimate of the total number of user stories for the project.

Total number of user stories for the project 𝑋 = 1 𝑚 𝑖=1 𝑚 𝑋 𝑖 = 1 𝑚 𝑖=1 𝑚 𝑆 𝑖 1 𝑛 (2) 𝑋 is an unbiased estimator of the total number of user stories for the project 𝑆 𝑖 is the number of user stories in the m-th epic m is the number of estimates done n is the number of epics in the project

Algorithm Divide the project scope into epics. Randomly sample one of the epics Analyze how many stories are in the sampled epic. Write down the number of stories. Using formula (1) calculate one estimate of the total number of stories for the project Repeat points 2-4 between 7 and 11 times Using formula (2) calculate the total number of stories for the project

Following is a calculation with data from a real ScrumDo. com project Following is a calculation with data from a real ScrumDo.com project. When the project finished in the backlog there were 29 epics and a total of 529 user stories.

Number of User Stories inside the epic Epic's selection probability Random epic selector Epic # Number of User Stories inside the epic Epic's selection probability Estimated total stories 0,733796 22 19 0,034483 551,00 0,596877 18 16 464,00 0,30461 9 24 696,00 0,988762 29 0,191704 6 11 319,00 0,184528 0,091998 3 20 580,00

Estimated project size Total Number of stories for the project Estimated project size 497 Number of RBS paths 7 SD 53 Median 551 Mode

LKCE2011 - Predictability & Measurement with Kanban by David Anderson In Kanban community we stand on the shoulders of giants and we build on top of existing practices. DJA’s approach is a subset of RBS.

DJA’s way to calculate the number of work items for a project? At 42:50 min DJA presented how to came out with the number of work items (user stories) for a project: “Randomly sample at least 7 and ideally 11 of the customer defined requirements and analyze them into work items that are meaningful to the development organization. User stories is just one example of a suitable work item type. Take the average number of the user stories per saga and multiply it with the number of sagas. The product is the number of user stories for the project.”

DJA’s formula compared with RBS – they are the same! 𝑆 = 𝑛 𝑚 𝑖=1 𝑚 𝑆 𝑖 = 1 𝑚 𝑖=1 𝑚 𝑆 𝑖 1 𝑛 “Randomly sample at least 7 and ideally 11 of the customer defined requirements and analyze them into work items that are meaningful to the development organization. User stories is just one example of a suitable work item type.” “Take the average number of the user stories per saga and multiply it with the number of sagas. The product is the number of user stories for the project.” “replace "user stories" with "work items suitable for the development organization."” http://www.lean-kanban-conference.de/what-happened-2011/predictability-and-measurement-with-kanban/ https://groups.yahoo.com/neo/groups/kanbandev/conversations/messages/16704 Where: 𝑆 is the number of work items per saga n is the number of sagas in the project m is the number of sagas broken down into user stories

Applying RBS for estimating total number of tasks in a project

When we size each user story in the number of tasks then the project size is the total of all tasks.

Tasks based sizing model Product … Epic A Epic N The delivery time per scenario will come from the historical delivery rate of the system. … User Story 1 User Story N … Task 1 Task N

… Task 1 Task N … User Story 1 User Story N … Epic A Epic N Product

Mapping Product Trunk Epic Branch User Story Terminal Shoot Number of tasks per User story Number of Fruit on the Shoot Mapping Product = Trunk Epic = Branch User Story = Terminal Shoot Number of tasks per User story = Number of Fruit on the Shoot

Estimate of the of total number of tasks for a project 𝑋 𝑖 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑎𝑠𝑘𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝑈𝑠𝑒𝑟 𝑠𝑡𝑜𝑟𝑦 1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐸𝑝𝑖𝑐𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡 1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑈𝑠𝑒𝑟 𝑠𝑡𝑜𝑟𝑖𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝐸𝑝𝑖𝑐 (3) Where: 𝑋 𝑖 is an unbiased estimator of the population total of the of story points for the project.

Total number of tasks for the project 𝑋 = 1 𝑚 𝑖=1 𝑚 𝑋 𝑖 (4) Where: 𝑋 is an unbiased estimator of the total number of tasks for the project. m is the number of estimates done

Algorithm Divide the project scope into epics. Randomly sample one of the epics Analyze how many stories are in the sampled epic. Write down the number of stories. Randomly sample one of the stories of the epic from p.2 Establish the tasks for the story from p.4 Using formula (3) calculate one estimate of the total number of tasks for the project Repeat points 2-6 between 7 and 11 times Using formula (4) calculate the total number of tasks for the project

Following is a calculation with data from a real ScrumDo. com project Following is a calculation with data from a real ScrumDo.com project. When the project finished in the backlog there were 15 epics, 720 user stories and a total of 5591 tasks.

Number of User Stories inside the epic Random story selector Random epic selector Epic # Number of User Stories inside the epic Random story selector Selected user story Tasks for the selected story Epic's selection probability Story's selection probability Conditional selection probability Estimated total tasks 0,887642 14 42 0,649722871 545769 12 0,066667 0,02381 0,0015873 7560,00 0,763994 51 0,017087888 506420 8 0,019608 0,0013072 6120,00 0,897303 0,571814178 541008 1 630,00 0,542088 9 37 0,559320969 544703 2 0,027027 0,0018018 1110,00 0,510646 48 0,360797137 527216 0,020833 0,0013889 10080,00 0,457058 7 46 0,892151817 564853 0,021739 0,0014493 9660,00 0,736139 0,972991924 567925 0,00

Estimated project size Total Number of tasks for the project Estimated project size 5023 Number of RBS paths 7 SD 1652 Median 6120

Applying RBS for estimating total number of BDD scenarios in a project

Scenario based sizing model Product … Epic A Epic N The delivery time per scenario will come from the historical delivery rate of the system. … User Story 1 User Story N … Scenario 1 Scenario N

… … … Scenario 1 Scenario N User Story 1 User Story N Epic A Epic N Product

What is a Scenario? A scenario is an acceptance test customers could understand written in their ordinary business language. It is a formal test conducted to determine whether or not the system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system. A User Story can have one or more scenarios

When we size each user story in the number of scenarios then the project size is the total of all scenarios.

Mapping Product Trunk Epic Branch User Story Terminal Shoot Number of Scenarios per User story Number of Fruit on the Shoot Mapping Product = Trunk Epic = Branch User Story = Terminal Shoot Number of Scenarios per User story = Number of Fruit on the Shoot

An estimate of the of total number of scenarios for a project 𝑋 𝑖 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑒𝑛𝑎𝑟𝑖𝑜𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝑈𝑠𝑒𝑟 𝑠𝑡𝑜𝑟𝑦 1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐸𝑝𝑖𝑐𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑝𝑟𝑜𝑗𝑒𝑐𝑡 1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑈𝑠𝑒𝑟 𝑠𝑡𝑜𝑟𝑖𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 𝐸𝑝𝑖𝑐 (5) Where: 𝑋 𝑖 is an estimate of the total number of scenarios for the project.

Total number of scenarios for the project 𝑋 = 1 𝑚 𝑖=1 𝑚 𝑋 𝑖 (6) Where: 𝑋 is an unbiased estimator of the total number of scenarios for the project. m is the number of estimates done

Algorithm Divide the project scope into epics. Randomly sample one of the epics Analyze how many stories are in the sampled epic. Write down the number of stories. Randomly sample one of the stories of the epic from p.2 Establish the scenarios for the story from p.4 Using formula (5) calculate an estimate of the total number of scenarios for the project Repeat points 2-6 between 7 and 11 times Using formula (6) calculate the total number of scenarios for the project

Conclusion RBS is a forecasting technique for sizing software projects without prior identification, analysis and sizing of every single user story. Project size may be measured in story points, number of tasks, BDD scenarios etc. By running RBS on past data from actual projects, we found that the RBS would have estimated the same size without all the usual effort. RBS helps us to reduce uncertainty regarding “how much” software needs to be developed when we have to make portfolio related decisions, provide quotations on prospect projects etc.