Evaluation training NHS NEW Devon CCG Transforming Community Services Pathways for Adults with Complex Needs My name is Mark Harding. I am the Clinical Procurement Manager for South West Commissioning Support and will be undertaking the Evaluation Training with you today. Each slide contains commentary giving further information and explanation. Please click the Sound logo in the bottom right hand corner to listen to the commentary

Introduction The aim of this session is to provide training to those undertaking evaluation exercises for the Transforming Community Services – Pathways for Adults with Complex Needs Most Capable Provider Assessment. South West Commissioning Support – Clinical Procurement (SWCS) is supporting NHS NEW Devon CCG to ensure a robust and standardised process is maintained and a good outcome achieved. The aim of this session is to provide training to all those undertaking the evaluation exercise for the Transforming Community Services – Pathways for Adults with Complex Needs Most Capable Provider Assessment. South West Commissioning Support – Clinical Procurement team is working with NHS NEW Devon CCG to ensure a robust and standardised process is maintained throughout.

Evaluation Principles (1) The principles behind evaluation are that the process and the outcome should be fair, open and transparent to potential providers. To facilitate this, the evaluation process should be documented, objective and robust. This: ensures that the process and methodology have been applied correctly and consistently demonstrates the evaluation principles have been upheld; and enables auditing of the result(s). It is good practice to adhere to the general obligations of transparency, equal treatment, non-discrimination and proportionality. Therefore the principles behind evaluation are that the process and the outcome should be fair, open and transparent to all the potential providers. To facilitate this, SWCS will ensure that the evaluation process is documented, objective and robust. This will: ensure that the process and methodology has been applied correctly and consistently demonstrate that the evaluation principles have been upheld; and enable auditing of the result(s).

Evaluation Principles (2) In the context of the assessment, all providers must be treated equally. SWCS will ensure that all evaluation processes in relation to this Project are well documented. In keeping with this principle, the evaluation should be considered absolute and not comparative. This means that each Provider/Response is assessed against a pre-determined standard (the questions asked) and not against each other. It is important to stress that the evaluation must be considered absolute and not comparative. This means that each Provider/Response is assessed against a pre- determined standard (the questions that have been asked) and not against each other.

Invitation to Propose a Solution for Pathways for People with Complex Needs The Invitation to Propose a Solution was sent out to those organisations that requested to be assessed as the most capable provider for each locality. Six questions have been asked investigating how the providers will deliver integrated, personal and sustainable care and support to patients in Devon. Each provider has been asked to answer these questions and submit their responses. Each element of the assessment has been weighted in line with the commissioner’s views on the risk and importance of the element. It is anticipated that the responses will be available for evaluators to view on 13th October 2014 once compliance checks have been completed. The Invitation to Propose a Solution was sent out to those organisations that requested to be assessed as the most capable provider for each locality. Six questions have been asked investigating how the providers will deliver integrated, personal and sustainable care and support to patients in Devon. Each provider has been asked to answer these questions and submit their responses. These questions have a word limit of 1500 words Each Question has been weighted in line with the commissioner’s views on the risk and importance of the element. However, we ask you as an evaluator to look at all questions with the same vigour. It is anticipated that the responses will be available for evaluators to view on 13th October 2014 once compliance checks have been completed. SWCS will send the resources and the template score sheet to each evaluator via e- mail.

Weightings Question Weighting Assessment Criteria 1 25% Providers that ensure care is delivered across care pathways and across organisational boundaries, delivering fully embedded solutions within the locality urgent care systems to meet patients’ needs and deliver the best clinical outcomes for the future 2 15% Deliver community services in a clinically and financially sustainable and improving manner, recognising the financially challenged economy status 3 Delivery in an integrated system that makes a step change beyond current integration, takes into account the changing landscape of health and social care commissioning, and includes and supports integrated health and social care delivery 4 Design and deliver services that meet the needs of patients in a high quality, safe manner and are easy for patients to use and understand 5 Delivery of one governance process working effectively within the locality urgent care system recognising that where pathways cross organisations it is imperative to maintain, develop and enhance any formal partnership arrangements 6 As a locality community delivery system leader fulfilment of responsibilities towards a consistent model and outcomes across the CCG in a 3 year period?

23nd September 2014 – 6th October 2014 Timetable Stage Due Date Solutions Document issued 22nd September 2014 Provider clarification stage 23nd September 2014 – 6th October 2014 Provider response deadline 12 noon on13th October 2014 Provider Response Compliance Checks 13th October 2014 Responses and Evaluation pack issued to Evaluators Score Sheets Returned by Evaluators 17th October 2014 Moderation Meeting 21st October 2014 Most Capable Provider Recommendation Report Submitted 22nd October 2014 Governing Body Meeting 5th November 2014

Evaluation (1) This is a screenshot of an example evaluation template scoresheet. The scoring criteria is stated at the top. This will be described in more detail on the next slide. The question number, weighting and actual question is stated. We ask evaluators to read the question and then the response given by the provider. The template is used to note what was good about the response and what was not so good. This will inform the score that you allocate in the final column. It is very important to give comments as these will be used to provide feedback to both successful and unsuccessful providers as required for transparency purposes. Please note the feedback given to providers will not attribute any comment to an evaluator. We will give the final moderated score to each relevant bidder with themes around what was good and what was not so good about their submission.

Evaluation (2) Evaluators are asked to score the response given to each set question as per the following scoring criteria: Assessment Score Interpretation Excellent 5 A comprehensive response but to a significantly better degree or likely to result in increased quality (including improvement through innovation) Good 4 A comprehensive response submitted in terms of detail and relevance Acceptable 3 An acceptable response submitted in terms of the level of detail, accuracy and relevance Minor Reservations 2 Limited information provided or a response that is inadequate or only partially addresses the question Major Reservations 1 Response to the question received does not address the question that was posed. Deficient Response to the question (or an implicit requirement) significantly deficient or no response received Evaluators are asked to score the response given to each set question as per the following scoring criteria: 0 – Deficient. This score should be allocated if you consider the response to the question (or an implicit requirement) given by the provider to be significantly deficient or no response received. 1 – Major Reservations. If you have Major reservations with the response as it does not address the question that was posed a score of 1 should be allocated. 2 – Minor Reservations. If you have minor reservations over the response that has been given with limited information provided, or a response that is inadequate, or only partially addresses the question, then a score of 2 should be given. 3 – Acceptable. If an acceptable response is submitted by the provider in terms of the level of detail, accuracy and relevance then a score of 3 should be given. 4 – Good. If a comprehensive response is submitted in terms of detail and relevance by the provider please allocate a score of ‘4’. 5 – Excellent. This score should be given if you consider the response given by the provider to be comprehensive but to a significantly better degree or likely to result in increased quality (including improvement through innovation)

Evaluation (3) The evaluation is reached by considering the offered response in the context of such factors as: the Provider’s understanding of the requirement documented evidence of past/transferable relevant experience and; other justifications for confidence in service standards including quality assurance, continual improvement and risk management. Factors to really consider when reviewing the responses include: the Provider’s understanding of the requirement documented evidence of past/transferable relevant experience and; other justifications for confidence in service standards including quality assurance, continual improvement and risk management.

Moderation The moderation meeting will take place on 21st October 2014 Each evaluator will have evaluated the criteria independently and placed comments and a score against each question within the evaluation template for each option. This score will be based on the scoring criteria and guidance provided in this training session. Evaluators should send their completed sheets to the procurement lead for consolidation by close of play on 17th October 2014. The aim of the moderation meeting is to gain a consensus view and provide a single score for each criterion, for each option, once views from fellow evaluators have been heard. Moderation meetings are used to gain a consensus view and provide a single score for each criterion, once views from fellow evaluators have been heard. The meeting is facilitated by the procurement lead. If there is a variety in scores allocated by the evaluators for a question a review will take place. Each evaluator will be able to reflect on the score they have allocated and take into account the views of the other evaluators. The aim is to obtain a single score for each question. The justifications for the final moderated score will be documented by the procurement lead.

Conclusion of Evaluation Training If there is any aspect of this training session that you would like further understanding of please contact: Garry Mitchell Associate Director of Procurement 07786 275 627 Garry.Mitchell@nhs.net Mark Harding Clinical Procurement Manager 07717 808 055 Mark.Harding@nhs.net

Conclusion of Evaluation Training Please complete and electronically sign the Declaration form Please send the form to Clinical.Procurement@swcsu.nhs.uk by 10th October