1. The Business of Plant Breeding: Monitoring, Evaluation and Learning
Market-led approaches to new variety design in Africa
Chapter 6
Monitoring, Evaluation and Learning
Jean Claude Rubyogo and Ivan Rwomushana

2. Chapter 6 Monitoring, Evaluation and Learning
Jean Claude Rubyogo
Seed Systems and Agricultural Technology Transfer, CIAT-Tanzania
and
Ivan Rwomushana
ICIPE, Nairobi, Kenya

3. Objectives
Performance benchmarking: To enable breeders to devise a realistic performance assessment plan for their demand-led breeding programme that:
(i) Is incorporated into strategies and stage plans for new variety design and engages with clients and stakeholders in the value chain; and
(ii) Develops key performance indicators tailored for new variety development and delivery of a demand-led breeding programme’s goals and objectives.

4. Objectives 2. Variety adoption and performance tracking
To enable breeders to appreciate the importance of variety adoption assessment in demand-led breeding
To design pathways for monitoring progress in demand-led breeding with value chain clients, with defined responsibilities for the various actors
To explore improved, low cost methods for variety tracking to assess adoption of new varieties

5. Chapter 6 Outline Introduction Performance Benchmarking
Principles and Best Practices in Demand-led Breeding
Monitoring, Evaluation and Learning for Demand-led Breeding
Variety Adoption and Performance Tracking
Case Study on Bean Improvement in Ethiopia
Learning and Communications

6. Opening Discussion (individual exercise and plenary discussion )
1. INTRODUCTION
Opening Discussion (individual exercise and plenary discussion )
Do you have institutional M&E systems in your organization?
Are you using it in your breeding programme?
How do you measure success of your breeding programme?
How do you get feedback from value chain clients and other stakeholders?

7. 2. PERFORMANCE BENCHMARKING

8. Definition of Terms
Breeding program: “The combination of all the activities required to create a portfolio of new varieties that serve farmers in different market segments and their value chains”
Breeding project: “The means for creating a new variety to serve the needs of farmers, their markets and value chains”

9. Definition of Terms
Breeding goal: “The overarching aim that guides decision making within the breeding project”
Breeding objectives: “the measurable steps that can be undertaken to deliver the goal”

10. Definition of terms SMART Breeding objectives are:
Specific: Target specific aspects or combination of traits required by clients or the value chain for improvement
Measurable: Can be quantified or have at least an indicator of performance
Achievable: Can be delivered within the given time, knowledge, resources and enabling environment
Relevant: Must relate closely and support the overall goal of the project
Time-related: Appropriate for the timescales possible and it can be specified when results will be available

11. Definition of Terms
Project Outputs: “The product(s) produced during the demand-led breeding project”
Project Outcome: “The changes in behaviour or events that take place as a result of the new variety having been designed, developed, released and made available to clients”
Project Impact: “The economic and social benefits that accrue to farmers, their communities and their crop value chains as a result of using the new variety (or varieties) created during a plant breeding project”

12. Definition of Terms
Key performance indicator: “A descriptor/metric that can be measured to provide a quantitative assessment of the performance of core components of the project, such as efficiency and effectiveness of operational delivery, project outputs, outcomes and impact”
Target: “A quantitative expression of the desired result of the breeding project. It can be used to describe an output, an outcome and be a key performance indicator”

13. Definition of Terms
 Variety branding: “A name, term, design, symbol or other feature that distinguishes one institution’s/company’s/ individual breeder’s variety from those of others”

14. 3. PRINCIPLES AND BEST PRACTICES in DEMAND-LED BREEDING

15. Principles of Demand-led Breeding:
Target driven: Demand-led breeding is target driven and this is embedded in breeding and related M&E process
Demand-led strategy: A demand-led development strategy is designed for each new variety. It includes the components of “why”, “what”, “who”, “when” and “how”
Performance indicators: The expected level of engagement and emphasis placed on the views of clients on the performance and use of varieties is much higher in demand-led than in conventional breeding programmes

16. Demand-led Breeding Stage Plan
Line progression decisions
Early Development
Discovery
Proof of Concept
Late Development
Pre-Commercial
Commercial
Discontinue
Variety Design A C D E F G H B
Market research.
Product profile creation.
Development Investment
case created.
Identification, characterization and selection of specific traits to be used
Line Development. Development and screening of product components or pre-products rather than products
Small plot screening of products as commercial prototypes
Product prototype characterization and selection based on performance in replicated trials and wider evaluation
Preparing for launch
Product sales
Launch
Rapid Growth Peak Sales
Sales Decline
Stop sales
Elimination of inventory
Breeding & Selection
Product Evaluation and Scale-up
Life Cycle Management
Investment Decision
Initiation of research on germplasm aligned to the product concept
Candidates
selected
requiring characterization, scaling-up and regulatory approval
Product
of specific products in specific market segments
Product discontinuation
Products that have completed discontinuation activities, and stock removed from inventories
Product concept
Discovery
Key
Stage Gate decision point
Stage
Stage activities
Phase
Modified version of Syngenta Seeds stage plan

17. Variety Branding
Variety brand: name, term, design, symbol or other feature that distinguishes one variety from those of others
Appears on seed packages or printed as flyers/ posters used for marketing
Ensures returns on investments for breeders
Enables breeders to be recognized for their skills and makes breeding a valued profession

18. 4. MONITORING, EVALUATION AND LEARNING FOR DEMAND-LED BREEDING

19. 4. Monitoring, Evaluation (M&E) and Learning for Demand-led Breeding
What are the differences between monitoring, evaluation and learning in current breeding programmes vs demand-led breeding projects and programmes?
What do these differences mean for your demand led-breeding programm, for M&E and learning systems and the possible implementation challenges?
What does this mean for your role as a breeder?

20. Monitoring, Evaluation (M&E) and Learning
Monitoring, evaluation and learning is a process to improve project performance in achieving results, and is applicable to any breeding programme.
Four reasons for using M&E and learning in demand-led breeding projects/programmes are:
1. Delivery
To support development and delivery of new varieties to meet clients’ needs, to their design specification, on time and budget
2. Quality
To provide data on project progress and effectiveness
To improve project management and decision making
To provide data to plan future resource needs
To provide data useful for policy making and advocacy

21. Monitoring, Evaluation (M&E) and Learning
3. Accountability
To ensure accountability to clients, investors, partners, value chain actors, other stakeholders
4. Learning
To provide opportunities to learn from experience of current breeding projects/programme
To provide evidence about what works and what does not, to inform future projects and scaling up

22. Monitoring
Monitoring is a continuous observation and checking procedure on the progress of on-going breeding activities
Compares progress of activities against milestones and timelines for decisions in the stage plan
Compares estimated and actual costs of the project against its approved budget
The purposes of monitoring are:
To support reaching the milestones and targets set for the project in a timely manner;
To determine if corrective action is required to solve emerging problems or any delays;
To identify improvements that need to be made to the stage plan.

23. Evaluation
Evaluation is a systematic, objective examination of the performance of a breeding project and delivery of its goals, objectives and targets
The best evaluations look at project performance in terms of relevance, effectiveness and efficiency and if expected outputs, outcomes and impact have been or will be achieved on time and on budget

24. Evaluation
Four key decision points in the stage plan when critical evaluation needs to be undertaken to support investment and progression decisions, are:
Investment decision: Decision to start a breeding project and invest in creating a new variety, based on a demand-driven product profile
Commercial candidates: Deciding on lead lines to be developed and scaled up
New variety release
Post-launch adoption and impact assessment

25. Key evaluation questions
Meeting trait performance targets: How close is the performance of the new variety to the new variety design targets set in the product profile? Specifically, are the genetic gains required for each of the priority traits being delivered?
Satisfying clients’ needs: Does the variety satisfy client’s needs and demand? Is it preferred to older varieties? Has it been adopted by the target numbers of farmers it was designed for?

26. Key evaluation questions
Impact: Does the variety create the economic, social and environmental impact at the individual, household and community level? (as defined in the benefits case used to justify the investment in the breeding project)
Impact may be assessed by ex post evaluation only several years after varietal release

27. How is M&E integrated into Demand- led Breeding Projects/Programme?
M&E is an important component for success and continuous improvement in demand-led breeding
M&E for demand-led breeding is designed to be primarily target driven (and not breeding activity driven).
Key performance indicators of success include metrics on:
Performance of new varieties, meeting trait specifications
Client satisfaction
Use of new varieties by farmers and their value chains

28. M&E Implications for the Breeder
Metrics: Breeders should support performance measurements that go beyond number of varieties registered and include metrics on product use and performance after release
Variety identification: Breeders should build identity recognition systems into their varieties to allow a simple, cost effective, and ideally in-situ identification to enable monitoring of variety adoption monitoring
Clients: Greater engagement of breeders with clients and value chains at the key decision points in the variety stage plan helps to ensure demand and uptake on release

29. Key performance indicators for SMART breeding objectives
The quality and effectiveness of breeding objectives can be improved by ensuring they have S.M.A.R.T characteristics:
Specific – target specific aspects or combination of traits required by clients or the value chain for improvement
Measurable – can be quantified or have at least an indicator of performance
Achievable – can be delivered within the given knowledge, resources and enabling environment
Relevant – must relate closely and support the overall goal of the project
Time-related – are appropriate for the timescales possible and it can be specified when results will be available

30. Ugandan dry bean seed value chain

31. Group Exercise – Formulation of KPIs
Using the Ugandan dry bean seed value chain:
What information would you want from each value chain actor to measure the success of your variety?
List the KPIs (quantitative and qualitative indicators) along this seed value chain

32. 5. VARIETY ADOPTION AND PERFORMANCE TRACKING

33. Variety Adoption and Performance Tracking
How do you know which is your seed/variety in the field?
Do you have a cost-effective method to track your new variety after release?

34. Variety Adoption and Performance Tracking
Success of a breeding programme can only be verified if the adoption pathways and performances of the products are tracked and documented
Breeders need to be aware of all variety identity and tracking technologies available e.g. phenotypic and low- cost molecular approaches
Evaluate each tracking option in terms of accuracy, technical feasibility and cost
Select and incorporate the most appropriate tracking methods into the development strategy and stage plan

35. Variety Tracking using GPS Coordinates
Important for breeders to:
Monitor the movement of their varieties
Determine if varieties are adopted and being used by the farmers.
Also include locations of seed sales agencies and their geographical distribution
New IT can help track seed sales of new varieties (e.g. GPS, mobile phones).
An example of variety tracking using GPS coordinates in six districts of Tanzania

36. 6. LEARNING AND COMMUNICATION

37. Learning and Communication
Learning is an important aspect of a breeding project that is often overlooked
Success stories/case studies are important communication tools to share learning and impacts amongst breeders, clients, investors and value chain actors
Case studies provide a medium for understanding and acting on knowledge gained and lessons learned
Case studies provide a lasting record of the breeding team’s accomplishments and impact well beyond the completion of a specific breeding project (e.g. Norman Borlaug (wheat) and Gibesa Ejeta (sorghum) effect)
Quantitative key performance measures – these are numerical and are gathered using e.g. surveys with set questionnaires, predefined data collection formats useful for recording and comparing predetermined variables
Qualitative descriptions – the change is shown through description. e.g. rapid assessment techniques, focus group discussions, and semi-structured interviews
Participatory methods – these include visual methods that do not rely on high literacy levels e.g. photos, diagrams, maps, timelines, stories, drama, song, mapping, ranking and scoring. These approaches allow the differences in people’s interests, needs and priorities to be recognized by insiders and outsiders, and form the basis for negotiation between stakeholders. They also allow people to benefit from analysing and asserting their own interests, and make a meaningful contribution to the way the project is designed, managed and implemented. It is a way of ensuring relevance of the project and promoting ownership of local people, which is essential for any chance of sustainability and success in the long term
Most significant change (MSC) – this is one of the most popular approaches that have been developed for monitoring and evaluation without indicators. It is a form of participatory monitoring and evaluation, based on telling stories about events people think were important and why they think they were significant. With this approach, there is no need to explain what an indicator is or learn special professional skills, so everyone can participate and the technique can be used in different cultural contexts. It is particularly suited to monitoring thea project where the focus is on learning rather than just delivery
Outcome mapping – this technique focuses on changes in the behaviour and attitudes of people, groups and organizations that are directly affected by the development or release of the new variety. E.g. outcome mapping focuses on the effect of the development project on the roles and participation of partners from the value chain for this and future projects

38. Learning and Communication
Lessons can also be learned from less successful breeding projects
Requires visionary R&D leadership to ensure negative evaluations lead to positive outcomes, by contributing to professional development of breeders and improving new variety designs for future breeding projects

39. Case study Beans in Ethiopia 2004-2014

40. Other Success Stories in Plant Breeding
Name of breeder, research institute/organization
Name a success story from a national/international breeding programme?
What is the name of the variety? When was it released?
Where was it released ? Can it be tracked via GPS/other?
What is the evidence of adoption by farmers? Data?
Does new variety meet and/or expand market demand?
What are quantifiable benefits/impact of the new, (demand-led) variety?

41. The Business of Plant Breeding: Monitoring, Evaluation and Learning
Market-led approaches to new variety design in Africa
Chapter 6
Monitoring, Evaluation and Learning
Jean Claude Rubyogo and Ivan Rwomushana