The Family Nurse Partnership (FNP) Programme Developing the Evaluation Framework

FNP Programme It is an intensive nurse-led home visiting programme that enables the family nurse to visit the same client from early pregnancy until the child is two It is an intervention for young, first time mothers, who meet the broad eligibility criteria (19 and under at LMP, keeping their baby, living within geographical boundaries, <28 wks gestation at recruitment) Programme aims: Improve pregnancy outcomes Improve child development Improve the economic self-sufficiency of the family

FNP in UK England since April 2007. Implemented in 55 LA/ PCT areas. FNP is now being tested in Scotland for the first time in NHS Lothian City of Edinburgh Community Health Partnership (Edinburgh CHP). First babies enrolled in March 2010. 2 teams in NHS Tayside, covering Dundee, Angus and Perth and Kinross CHP’s. England: Formative evaluation of the first ten sites reported 2011 England: RCT in 18 sites which will report, initially, in 2013. The RCT will assess what the benefits and costs associated with FNP, looking closely at prenatal health behaviours, and early child health outcomes.

FNP Evaluation Evaluation Tem: Scottish Centre for Social Research The overall aim: evaluate the implementation of the programme in Scotland (Lothian) Specific questions: Is the programme being implemented as intended? If not, why not? How does the programme work in Scotland (Lothian)? Wider implications for implementing the programme in Scotland. Evaluation implementation: Monitoring and Evaluation Framework: Internal (FNP data) and External (stakeholder interviews, qualitative panel and focus groups) relevant to outcomes of interest in Scotland Implementation issues: Identify the factors which support or inhibit the delivery of the programme. Report family nurse experience of implementing the programme, including recruitment, training and retaining and structural support to dedicated Family Nurse for the duration of the programme. Report on the client experience of the programme including the acceptability and response to the programme.

How we developed M+E framework Series of focused meetings facilitated to achieve consensus on the what the programme was intending to achieve (outcomes) and how this was intended to operate (processes and assumptions) Produced two logic models: a Google Earth view and an implementation model These provided framework for the M+E

Preliminary logic model using EB and incorporating outcomes in service level agreement

FNP: High level strategic model

FNP: Implementation model

FNP: Embedded implementation model

From Having to Using…… We used the logic models to frame monitoring and evaluation of the programme by: Prioritising key outcomes and assumptions of interest Prioritise key questions addressing above Agreeing who would collect and analyse which data, when and how Decisions underpinned by considerations of feasibility, acceptability and data robustness (including how to improve these)

Box code (from logic model) 3 and links to 1 and 2 Logic If the team attend training and are supervised, then they will possess requisite competencies Question Does team receive the training & support intended & develop req’d knowledge/ skills? Indicator(s) Who’ll collect data? Is this a fidelity req’t? Who’ll analyse? Any additional considerations? Proportion of team attending each mandatory course* % of learning events run* Self reports of satisfaction and perceived utility/effectiveness of training and learning events. Frequency of supervision sessions* e.g. for each FN, no of weeks per quarter that timetabled supervision takes place, expressed as a proportion of working/available weeks ie excludes sickness absence % of required accompanied visits that take place*e.g. no of accompanied visits per FN per 4 months and % of FNs who receive min quota of accompanied visits every 4 months. Self-reports of feasibility of roles and competence to deliver it (based on job spec and on practice); also whether/how psychologist support worked in practice Internal External (interviews with all FNs and supervisor) External (interviews with FNP) FR: Attendance at 4 residential training courses FR: supervisor runs pre and post learning events FR for weekly supervision FR for each FN to be accompanied at least once every 4 months FNP ScotCen It will be imperative that the FNP builds in a process of regular review (every 3 months?) in order to address any shortfalls in delivery req’ts. There would need to be some agreement about what qualifies as supervision e.g. a quick catch up in the corridor? All measures on this page will require good record keeping, submission of records to the administrator within a workable timescale We have assumed that the FN’s and supervisor meet person spec in job desc and so this does not need monitored

Box code (from logic model) n/a Assumption Attrition will be low Logic The project is only viable if most families participate AND If families find the support useful, they will stay engaged Question Does project meet the fidelity targets for attrition? Indicator(s) Who will collect data: internal or external evaluation? Is this a fidelity requirement/ goal? Who’ll analyse the data in the first instance? Any additional considerations? Percentage leaving/dropping out of programme* Calculated as total no having left the programme divided by no enrolled. Implement programme alerts at monthly intervals if feasible. Internal via UK004B FR/G: Cumulative prog attrition is 40% or less thro to the child’s 2nd birthday and is10% or less during pregnancy….. FNP But included in ScotCen reports

Box code (from logic model) n/a Assumption Attrition will be low Logic The project is only viable if most families participate AND If families find the support useful, they will stay engaged Question Does project meet the fidelity targets for attrition? Indicator(s) Who will collect data: internal or external evaluation? Is this a fidelity requirement/ goal? Who’ll analyse the data in the first instance? Any additional considerations? Percentage leaving/dropping out of programme* Calculated as total no having left the programme divided by no enrolled. Implement programme alerts at monthly intervals if feasible. Acceptability and perceived utility of FN support Internal via UK004B External via interviews with clients/families Incl (if possible) a sample of those who drop out/leave) FR/G: Cumulative prog attrition is 40% or less thro to the child’s 2nd birthday and is10% or less during pregnancy….. FNP But included in ScotCen reports Form UK004B makes no provision for client leaving because they did not like the programme. Suggest that some extra fields are added to cover broader range of possible reasons for leaving

The M+E framework in action… Outcome: Mother takes good care of self Logic: If mother takes good care of self, the risk factors for the infant are reduced Question: Is there evidence that the FNP results in improved knowledge /health behaviours in clients prior to/following birth of baby? Indicator: Clients’ accounts of what they have learned about risk/protective factors Topic guide: Have you and your family nurse talked about smoking? What about drinking alcohol? Taking drugs? The food you should eat or not eat during your pregnancy? Keeping the baby safe? Interview responses Well I, to be honest I already knew about like drinking alcohol and taking drugs but I never knew about the smoking thing because my gran smoked with all her three kids while she was pregnant and my gran keeps on saying that later on they were fine but.. …because during my past pregnancy I was actually getting ready to drink alcohol again and she convinced me not to because I make a good home for the baby and the baby’s depending on me and stuff like that

Ist report: intake and early pregnancy Application Scottish context: ensures evaluation relevant to outcomes of interest in Scotland Evaluation tools: Identifies data collection relevant to outcomes of interest Wider work: informed NHS Lothian on wider maternity services work Ist report: intake and early pregnancy http://www.scotland.gov.uk/Publications/2011/07/28142203/0 Contact: Vikki Milne, victoria.milne@scotland.gsi.gov.uk