1. The Family Nurse Partnership (FNP) Programme Developing the Evaluation Framework

2. FNP Programme
It is an intensive nurse-led home visiting programme that enables the family nurse to visit the same client from early pregnancy until the child is two
It is an intervention for young, first time mothers, who meet the broad eligibility criteria (19 and under at LMP, keeping their baby, living within geographical boundaries, <28 wks gestation at recruitment)
Programme aims:
Improve pregnancy outcomes
Improve child development
Improve the economic self-sufficiency of the family

3. FNP in UK England since April 2007. Implemented in 55 LA/ PCT areas.
FNP is now being tested in Scotland for the first time in NHS Lothian City of Edinburgh Community Health Partnership (Edinburgh CHP). First babies enrolled in March 2010.
2 teams in NHS Tayside, covering Dundee, Angus and Perth and Kinross CHP’s.
England: Formative evaluation of the first ten sites reported 2011
England: RCT in 18 sites which will report, initially, in The RCT will assess what the benefits and costs associated with FNP, looking closely at prenatal health behaviours, and early child health outcomes.

4. FNP Evaluation Evaluation Tem: Scottish Centre for Social Research
The overall aim: evaluate the implementation of the programme in Scotland (Lothian)
Specific questions:
Is the programme being implemented as intended? If not, why not?
How does the programme work in Scotland (Lothian)?
Wider implications for implementing the programme in Scotland.
Evaluation implementation: Monitoring and Evaluation Framework: Internal (FNP data) and External (stakeholder interviews, qualitative panel and focus groups) relevant to outcomes of interest in Scotland
Implementation issues:
Identify the factors which support or inhibit the delivery of the programme.
Report family nurse experience of implementing the programme, including recruitment, training and retaining and structural support to dedicated Family Nurse for the duration of the programme.
Report on the client experience of the programme including the acceptability and response to the programme.

5. How we developed M+E framework
Series of focused meetings facilitated to achieve consensus on the what the programme was intending to achieve (outcomes) and how this was intended to operate (processes and assumptions)
Produced two logic models: a Google Earth view and an implementation model
These provided framework for the M+E

6. Preliminary logic model using EB and incorporating outcomes in service level agreement

7. FNP: High level strategic model

8. FNP: Implementation model

9. FNP: Embedded implementation model

10. From Having to Using……
We used the logic models to frame monitoring and evaluation of the programme by:
Prioritising key outcomes and assumptions of interest
Prioritise key questions addressing above
Agreeing who would collect and analyse which data, when and how
Decisions underpinned by considerations of feasibility, acceptability and data robustness (including how to improve these)

11. Box code (from logic model) 3 and links to 1 and 2 Logic
If the team attend training and are supervised, then they will possess requisite competencies
Question
Does team receive the training & support intended & develop req’d knowledge/ skills?
Indicator(s)
Who’ll collect data?
Is this a fidelity req’t?
Who’ll analyse?
Any additional considerations?
Proportion of team attending each mandatory course*
% of learning events run*
Self reports of satisfaction and perceived utility/effectiveness of training and learning events.
Frequency of supervision sessions*
e.g. for each FN, no of weeks per quarter that timetabled supervision takes place, expressed as a proportion of working/available weeks ie excludes sickness absence
% of required accompanied visits that take place*e.g. no of accompanied visits per FN per 4 months and % of FNs who receive min quota of accompanied visits every 4 months.
Self-reports of feasibility of roles and competence to deliver it (based on job spec and on practice); also whether/how psychologist support worked in practice
Internal
External (interviews with all FNs and supervisor)
External (interviews with FNP)
FR: Attendance at 4 residential training courses
FR: supervisor runs pre and post learning events
FR for weekly supervision
FR for each FN to be accompanied at least once every 4 months
FNP
ScotCen
It will be imperative that the FNP builds in a process of regular review (every 3 months?)
in order to address any shortfalls in delivery req’ts.
There would need to be some agreement about what qualifies as supervision e.g. a quick catch up in the corridor?
All measures on this page will require good record keeping, submission of records to the administrator within a workable timescale
We have assumed that the FN’s and supervisor meet person spec in job desc and so this does not need monitored

12. Box code (from logic model) n/a Assumption Attrition will be low Logic
The project is only viable if most families participate
AND
If families find the support useful, they will stay engaged
Question
Does project meet the fidelity targets for attrition?
Indicator(s)
Who will collect data: internal or external evaluation?
Is this
a fidelity requirement/
goal?
Who’ll analyse the data in the first instance?
Any additional considerations?
Percentage leaving/dropping out of programme*
Calculated as total no having left the programme divided by no enrolled.
Implement programme alerts at monthly intervals if feasible.
Internal via UK004B
FR/G:
Cumulative prog attrition is 40% or less thro to the child’s 2nd birthday
and is10% or less during pregnancy…..
FNP
But included in ScotCen reports

13. Box code (from logic model) n/a Assumption Attrition will be low Logic
The project is only viable if most families participate
AND
If families find the support useful, they will stay engaged
Question
Does project meet the fidelity targets for attrition?
Indicator(s)
Who will collect data: internal or external evaluation?
Is this
a fidelity requirement/
goal?
Who’ll analyse the data in the first instance?
Any additional considerations?
Percentage leaving/dropping out of programme*
Calculated as total no having left the programme divided by no enrolled.
Implement programme alerts at monthly intervals if feasible.
Acceptability and perceived utility of FN support
Internal via UK004B
External via
interviews with clients/families
Incl (if possible) a sample of those who drop out/leave)
FR/G:
Cumulative prog attrition is 40% or less thro to the child’s 2nd birthday
and is10% or less during pregnancy…..
FNP
But included in ScotCen reports
Form UK004B makes no provision for client leaving because they did not like the programme. Suggest that some extra fields are added to cover broader range of possible reasons for leaving

14. The M+E framework in action…
Outcome:
Mother takes good care of self
Logic:
If mother takes good care of self, the risk factors for the infant are reduced
Question:
Is there evidence that the FNP results in improved knowledge /health behaviours in clients prior to/following birth of baby?
Indicator:
Clients’ accounts of what they have learned about risk/protective factors
Topic guide:
Have you and your family nurse talked about smoking? What about drinking alcohol? Taking drugs? The food you should eat or not eat during your pregnancy? Keeping the baby safe?
Interview responses
Well I, to be honest I already knew about like drinking alcohol and taking drugs but I never knew about the smoking thing because my gran smoked with all her three kids while she was pregnant and my gran keeps on saying that later on they were fine but..
…because during my past pregnancy I was actually getting ready to drink alcohol again and she convinced me not to because I make a good home for the baby and the baby’s depending on me and stuff like that

15. Ist report: intake and early pregnancy
Application
Scottish context: ensures evaluation relevant to outcomes of interest in Scotland
Evaluation tools: Identifies data collection relevant to outcomes of interest
Wider work: informed NHS Lothian on wider maternity services work
Ist report: intake and early pregnancy
Contact: Vikki Milne,