Skip to main content

British Columbia Healthy Connections Project process evaluation: a mixed methods protocol to describe the implementation and delivery of the Nurse-Family Partnership in Canada

Abstract

Background

The Nurse-Family Partnership is a home visitation program for first-time, socially and economically disadvantaged mothers. The effectiveness of this public health intervention has been well established in the United States; however, whether the same beneficial outcomes will be obtained within the Canadian context is unknown. As part of the British Columbia Healthy Connections Project, which includes a trial comparing Nurse-Family Partnership’s effectiveness with existing services in British Columbia, we are conducting a process evaluation to describe and explain how the intervention is implemented and delivered across five regional Health Authorities.

Methods

A convergent parallel mixed methods research design will be used to address the process evaluation objectives. The principles of interpretive description will guide all sampling, data collection and analytic decisions in the qualitative component of the study. The full population of public health nurses and supervisors (n = 71) will discuss their experiences of implementing and delivering the program in interviews (or focus groups). Managers (n = 5–15) responsible for this portfolio will also be interviewed annually. Fidelity reports with quantitative data on the reach and the dose of the intervention will be collected and analyzed. Summaries of team meetings and supervisory sessions will be analyzed. Data will be used to compare, corroborate and explain results and variances across the five regional Health Authorities.

Discussion

The process evaluation results will be of immediate instrumental use to the program implementers to inform intervention delivery. Findings will contribute to the emerging body of evidence surrounding: 1) professional nurse home visitation practice issues; 2) best practices for meeting the needs of families living in rural and remote communities; 3) a deeper understanding of how health and social issues such as mental health problems including substance misuse and exposure to intimate partner violence affect a young mother’s capacity to parent; and 4) strategies to support professionals from the primary care, public health and child welfare sectors to work collaboratively to meet the needs of children and families who are at risk or experiencing maltreatment.

Peer Review reports

Background

Nurse-Family Partnership

The Nurse-Family Partnership (NFP) is a home visitation program for first-time, socially and economically disadvantaged pregnant women and mothers with young children. The overarching goals of this evidence-based public health intervention are to improve: 1) pregnancy outcomes; 2) child health and development; and 3) parents’ economic self-sufficiency [1]. Over the past three decades in the United States (US), the effectiveness of the NFP has been tested in three randomized controlled trials (RCTs) with diverse populations of American families living in unique geographic locations: Elmira, New York, Memphis, Tennessee, and Denver, Colorado [2]. Across these trials, consistent and enduring effects of the NFP have been established, including demonstrated improvements in prenatal health behaviours, birth outcomes, and child mental health and developmental outcomes [2]. Findings from the Memphis RCT also conclude that participation in NFP reduces preventable death among nurse-home visited mothers and their children [3]. Furthermore, NFP has been established as the intervention with the best evidence for preventing child maltreatment and other associated outcomes, including reductions in child injuries and emergency room visits [4]. Given the level of evidence attributable to this nurse home visitation intervention in improving the lives of disadvantaged children and families, extensive interest has been expressed by societies outside the US in implementing the NFP. Ongoing evaluation and international NFP research is coordinated by Dr. David Olds (the developer of the NFP) and his team at the Prevention Research Center for Family and Child Health, University of Colorado Denver. In countries outside of the US, there is considerable geographic, demographic, socio-economic, public policy, nursing workforce, and healthcare system variation; it therefore cannot be assumed that the impact of the NFP seen in the US trials will be the same in other countries. Dr. Olds has developed a four-phase model for adapting and evaluating the NFP in international contexts: 1) adaptation: exploration of adaptations needed to deliver the program in local contexts while ensuring fidelity to the 18 core elements [5] of the NFP model; 2) feasibility and acceptability pilot studies: initial small-scale assessment and implementation studies to inform what additional program adaptations may be needed to meet the needs in the new country’s context; 3) RCT: expansion and testing of the intervention in a large-scale RCT to determine NFP’s effectiveness in the specified context; and 4) continued refinement and expansion: if the RCT produces outcomes of public health significance, decisions can then be made regarding wide-scale expansion of the NFP program within the new country [6]. The adaptation of the NFP in England (known there as the Family-Nurse Partnership) is a prime example of how this process for adapting existing evidence-based interventions for new societies can be implemented. In England, the feasibility of implementing this home visitation program along with estimating program costs and determining program impact on key stakeholders was evaluated prior to initiation of the RCT to determine overall effectiveness [7].

Canadian evaluation of the Nurse-Family Partnership

The first two steps of the process for international NFP replications and evaluation were completed in Hamilton, Ontario, Canada. A concurrent mixed methods pilot study was conducted from 2008 to 2012 to determine overall feasibility and acceptability of the NFP program in Canada with 108 women who met the study eligibility criteria [8, 9]. Based on this work, to successfully deliver NFP in Canada, minor adaptations were made to the US NFP model elements. A list of the model elements to guide subsequent implementations in Canada is provided in Table 1.

Table 1 Canadian NFP model elements

Given the findings that the NFP was both feasible to deliver and acceptable to stakeholders in a predominately urban Canadian setting, the McMaster research team sought out provincial partners interested in participating in Phase 3 – the conduct of an RCT to determine overall effectiveness of the NFP to improve maternal and child health outcomes in a Canadian context [10]. As the NFP aligned with key mental health strategic plans and core public health programs, the province of British Columbia (BC) emerged as ready and committed to conducting the RCT. The BC Healthy Connections Project (BCHCP) is the resultant scientific evaluation that includes both an RCT to measure NFP’s effectiveness compared to existing services, and a process evaluation to describe how the NFP is implemented and delivered across the five participating BC Health Authorities (HAs), including describing variances within and between sites [11]. The BCHCP includes one final third project - the Healthy Foundations study that is focused on exploring the biological mechanisms that link intervention and behavioural outcomes in children. This paper outlines the BCHCP process evaluation methods; the BCHCP RCT protocol is the subject of a separate manuscript.

Process evaluations of complex public health interventions

As a complex intervention, the NFP has multiple interacting components, a range of diverse prenatal, child and maternal outcomes to influence, flexibility in intervention delivery, and sensitivity to local organizational and contextual factors [12]. For example, although PHNs have guidelines on when to conduct home visits, the timing and frequency of home visits will be influenced largely by individual client needs, availability, and engagement in the program. At the level of the individual HAs who are responsible for implementing and delivering the NFP as part of the BCHCP RCT intervention arm and process evaluation, organizational factors such as the level and quality of nursing supervision provided, the time allocated for PHNs to deliver the NFP (e.g., full-time or part-time), and administrative supports and resources for the BCHCP in general have the potential to influence fidelity to the program model. Each HA is also unique in its geography, population characteristics, and the pre-existing culture of partnerships and collaborations that are needed to support and sustain NFP at the community level. This level of complexity creates challenges for evaluating the effectiveness of the intervention and for understanding how the causal mechanisms of the intervention may influence intended outcomes [11].

Increasingly, process evaluations are carried out as part of the overall approach to conducting a comprehensive evaluation of a complex intervention [13]. A process evaluation can help to answer how and why an intervention succeeds, or fails, within different contexts. The primary functions of the BCHCP process evaluation will be to describe stakeholders’ perceptions and experiences of the intervention components, to assess if the components are delivered with fidelity, to describe planning, implementation and delivery processes and to provide a detailed description of a range of contextual factors that might explain variances across the five HAs. Process evaluations of public health interventions typically include measurements of: fidelity (quality); dose of the intervention delivered (completeness) and received (exposure, including engagement and satisfaction with intervention materials); reach (participation rate); recruitment (enrolment and retention); implementation; and context (individual, organizational, cultural, social, or environmental influences on implementation) [14, 15]. In evaluating a complex intervention, the process evaluation is ideally planned and implemented alongside an RCT. In BC, the NFP will be delivered by 5 unique HAs –our rationale for conducting the process evaluation was so that the different ways in which this intervention is implemented and delivered can be documented and analyzed. This contextual data will be essential for eventually understanding and explaining any potential inter-site variances that emerge from the trial. Data from this process evaluation will also make a substantial contribution internationally related to the adaptations required to successfully deliver NFP to families living in rural or remote communities. In addition to enhancing our understanding of the process outcomes related to delivery of the NFP, the data collected within the process evaluation will also enrich our understanding of a wide range of professional nursing practice and education issues.

Objectives of the BCHCP process evaluation

The overarching objectives for the BCHCP process evaluation are:

  1. 1.

    To determine the extent to which the NFP is delivered with fidelity to the 18 required Canadian model elements (Table 1).

  2. 2.

    To measure the dose of NFP (delivered and received), reach (participation rate throughout pregnancy, infancy and toddlerhood), and recruitment and retention.

  3. 3.

    To explore the acceptability of the NFP intervention to PHNs, supervisors (including the NFP Provincial Coordinator) and public health managers.

  4. 4.

    To describe PHNs’ and supervisors’ experiences of the NFP education program and to identify additional content areas needed for further knowledge and skill development.

  5. 5.

    To explore processes used to support NFP PHNs and supervisors through activities including reflective supervision, coaching and mentoring.

  6. 6.

    To identify contextual factors (individual, organizational, cultural, social and geographic) that influence: organizational adoption and implementation of the NFP; utilization of NFP visit-to-visit guidelines; caseload coordination; location, engagement and retention of NFP clients; sustainability of the NFP in BC; and the convening and sustaining of community advisory boards.

  7. 7.

    To identify adaptations to the NFP model elements to support PHNs and supervisors to meet the needs of clients living in smaller suburban, rural and remote communities.

  8. 8.

    To identify and describe PHNs’ experiences of delivering the NFP to clients and their families exposed to mental health problems including substance misuse, intimate partner violence (IPV), or engagement with the child welfare system.

Methods

Public health intervention process evaluation framework

The framework for conducting process evaluations of public health interventions developed by Linnan and Steckler [14] and operationalized by Saunders, Evans, and Joshi [15] was used to guide the planning, development and conduct of this study. Core components of the framework, and how they have been operationalized for the BCHCP process evaluation, are summarized in Table 2.

Table 2 BCHCP process evaluation framework

Design

Within the overarching BCHCP, the process evaluation has been systematically embedded throughout the full conduct of the RCT. A convergent parallel mixed methods [16] research design will be used to address the process evaluation objectives outlined above. This mixed methods design is characterized by the collection and analysis of different –both quantitative and qualitative – but complementary data. Qualitative data from program documents as well as interviews and focus groups with PHNs, supervisors (data from the NFP Provincial Coordinator will be coded and categorized with data from supervisors to ensure anonymity and confidentiality) and senior public health managers will be analyzed to explore and describe how the NFP is implemented and delivered across five diverse regional HAs. Concurrent with this data collection, NFP fidelity reports with quantitative data on the reach and the dose of the NFP intervention will be collected and analyzed. Summaries of the content and number of NFP site team meetings and supervisory sessions will also be provided to the research team for analysis. The reason for collecting both quantitative and qualitative data is to bring together the strengths of both research traditions to compare, corroborate and explain results and variances across the five HAs.

All sampling, data collection and data analysis decisions within the qualitative component of the process evaluation will be guided by the principles of interpretive description [17], an applied qualitative methodology that provides a structure for answering questions that arise within clinical practice [17]. Use of interpretive descriptive methods will enable us to provide a comprehensive description of how the NFP is implemented in BC, and to discover the associations, relationships and patterns among key concepts that are related to the implementation and delivery of this early intervention program, across the HAs. This qualitative approach is one that is used to address issues that have disciplinary relevance (i.e., public health nursing) and to address very specific practice goals. In this study, this approach will thus allow us to expand and extend what is already known about professional practice issues within the disciplinary field of nurse home visitation. Within the process evaluation, we are specifically seeking to understand the practice issue of how the NFP is implemented and delivered within the Canadian context, and more particularly, how this home visiting intervention may be adapted for delivery in smaller suburban, rural and remote areas compared to urban centres.

Settings

In BC, the Ministry of Health collaborates with the regional HAs that are responsible for the provision of all public health and healthcare services to individuals living within their boundaries. Through the BCHCP, each participating HA is also now responsible for the implementation and delivery of the NFP through their public health services. Four of the five HAs are participating in both the process evaluation and the RCT, while one HA (Northern Health Authority) is only participating in the process evaluation due to its remote geography.

Each HA is further divided into local health areas; the BC Ministry of Health and the participating HAs have identified the particular local health areas that will participate in the BCHCP. In local health areas where it was determined that the conditions were similar to the Hamilton, Ontario context where the pilot study was conducted – namely, that these sites comprise more densely populated large or medium sized urban communities, women who meet the program eligibility criteria will be invited to participate in the RCT. For the purpose of distinction, PHNs delivering the NFP to clients in these local health areas involved in the RCT are considered to be a part of the expanded process evaluation.

This distinction was necessary within the four HAs where the RCT is being conducted as there are a number of smaller locations where the BCHCP Scientific Team has determined that it was not feasible to collect RCT data due to either: a) low annual birth rates leading to inadequate sample size; or b) HA preferences. PHNs delivering the NFP in these smaller local health areas are part of the basic process evaluation. The fifth HA that is not enrolling any clients in the RCT will also be included in the basic process evaluation.

Some of the HA sites participating in the basic process evaluation are hypothesized to be considerably different compared to the predominantly urban setting where the NFP was first piloted in Hamilton. These HA sites are unique due to: geography (delivery of the program to families living in rural and remote communities); organizational structure (PHNs may not have any other NFP colleagues in the same office; supervisors and PHNs may not be co-located within the same offices); and PHN assignments (they may be allocated to deliver NFP as part of a series of public health responsibilities compared to many of the PHNs in larger sites who will work full-time on the NFP). Thus a significant focus of the basic process evaluation will be to determine what adaptations are required to the NFP model of home visitation to meet the needs of families in smaller suburban, rural and remote communities and to identify organizational and professional practice support mechanisms required by the PHNs.

Sample

Primary data will be collected from all BC NFP PHNs and supervisors (including the NFP Provincial Coordinator). These individuals will be eligible to participate in the process evaluation (basic or expanded) if they: 1) have completed, or are in the process of completing, the NFP education; 2) are delivering the NFP intervention to enrolled participants in the BCHCP (RCT intervention arm or process evaluation); and 3) who speak English. A purposeful sample of senior public health managers from each HA, who have the NFP program within the portfolio of services for which they are responsible, will also be invited to participate. Informed consent will be sought from all NFP service providers eligible to participate in the process evaluation. The total sample size at the commencement of the study is 71 participants including: supervisors + NPF provincial coordinator (n = 11) and PHNs (basic process evaluation n = 13, expanded process evaluation n = 47) (Fig. 1). Through a process of snowball sampling, we estimate inviting 5–15 senior public health managers (1–3 per participating health authority) to also participate. We anticipate that the sample size will increase during the study as new PHNs or supervisors hired to deliver the NFP during the participant recruitment period of the BCHCP will be invited to participate in the process evaluation following the completion of their NFP core education.

Fig. 1
figure1

Pursposeful sampling framework for the qualitative arm of the BCHCP process evaluation

Qualitative data collection

The process evaluation will be characterized by the triangulation of multiple data sources and types to address each research objective. A summary of data sources, types and the frequency of data collection for both the qualitative and quantitative components are summarized in Table 3. Using the principles of interpretive description, qualitative data will be collected from each consenting participant. The NFP PHNs, supervisors and managers will be invited to participate in either an in-depth 1:1 interview (basic process evaluation) or focus groups (expanded process evaluation). Individual interviews will be conducted at regular intervals (approximately every 6 months) until 2018, for a total of eight interviews each. At this rate, with a baseline sample of 11 supervisors (+provincial coordinator), a total of 88 interviews will be conducted with this group, and a total of 104 1:1 interviews with the 13 PHNs (basic process evaluation). However, senior public health managers will be invited to participate in only one annual interview (for a maximum of five interviews per individual; a total of 25–75 interviews across the duration of the study). Multiple interviews are required to document the emerging and ongoing procedures and contextual factors influencing both intermediate and long-term program delivery outcomes. Each participant will be asked to complete a short demographic questionnaire at baseline that will be integrated into the interview facilitation guide and then reviewed for changes during subsequent interviews.

Table 3 Process evaluation data sources, types and frequency of collection

The content of each interview has been informed by the study objectives – a list of core topics to be explored across the different phases of the study is summarized in Table 4. In-depth 1:1 interviews with supervisors, managers, the NFP provincial coordinator and those PHNs in the basic process evaluation will allow us to collect more nuanced data about adaptations required to deliver the NFP model in a range of geographic contexts. Through the use of interviews, we will be able to explore PHNs’ individual experiences of delivering the NFP and obtain a rich, detailed description of their overall experiences and perceptions with the BCHCP. Each semi-structured 1:1 interview will last approximately 60 min and will be conducted by the Research Coordinator via telephone at a mutually negotiated time. A copy of the interview guide will be shared with each participant prior to the interview, so that the individual can reflect on the questions and develop rich, detailed responses and case examples to share during the interview. Field notes to document observations noted during the interviews, emerging themes and issues, as well as topics to explore in subsequent interviews will be maintained by the interviewer and the research team.

Table 4 Summary of primary content – 1:1 interviews & focus groups

PHNs enrolled in the expanded process evaluation will similarly be invited to share their experiences and perceptions of the implementation and delivery of the NFP through focus groups conducted within each HA approximately every 6 months. This will provide us with the opportunity to identify unique contextual factors that influence program delivery between and across the four HAs that are also participating in the RCT. Each focus group will include 5–10 participants. Five focus groups will be held in each phase. Over the length of time of the expanded process evaluation, individual PHNs will participate in a total of eight focus groups (overall total number of focus groups for duration of project n = 40). The lead researcher who has over 16 years of experience conducting qualitative research (SMJ), or the Research Coordinator (with 8 years of qualitative experience) will facilitate all focus groups, which will be conducted on-site. The lead researcher will be responsible for the main facilitation of the group interviews and the Research Coordinator will manage the logistics and will maintain observational field notes. To minimize the risks of “group think” where participants shape their answers to conform with more dominant or powerful members of the group a range of strategies for posing the questions in different ways to elicit a range of responses will be used in each group [18]. PHNs who are unable to attend the focus group will be invited to complete an abbreviated 1:1 telephone interview.

Concurrent to the focus groups, field site visits will be completed across all five HAs. These field site visits will provide an opportunity to document observations about implementing agencies (i.e., HAs) [14] and provide the researchers with opportunities to develop rapport with study participants in the process evaluation, particularly the NFP supervisors and PHNs. Observations will also be captured in study field notes. Dr. Jack has successfully used this approach in her process evaluation documenting the implementation of the NFP IPV intervention within the context of an RCT to evaluate the effectiveness of the NFP IPV intervention [19] in the US. Initial and subsequent in-person on-site visits to meet key site leaders (e.g., managers, supervisors) and PHNs is anticipated to promote rapid engagement when subsequent data are collected via distance using teleconference, or webinar technology.

Key documents will be collected and analyzed in this process evaluation to corroborate and augment evidence from other data sources and to clarify terms and concepts that might have been mentioned in an interview. A review of documents might identify new issues for exploration in subsequent interviews or focus groups. The following documents will be collected on a quarterly basis by each NFP supervisor: 1) team meeting and case conference summary forms, and 2) a summary of weekly supervision records.

Quantitative data collection

The BCHCP Scientific Team will be summarizing and analyzing data from NFP program fidelity reports (aggregate data) provided by the BC Ministry of Health. Data contained within these fidelity reports are entered at the local health area level into a health surveillance software application. Data will be reported by implementation location (local health area) and by Health Authority. However, if at any one local unit level, the numbers are small, the Ministry of Health may choose to aggregate the data at the larger HA level only. The following information will be collected and analyzed from the NFP Program Fidelity Reports: 1) stage of pregnancy at time of enrolment; 2) percentage of time spent on each of the six domains during each of the three phases of the NFP program (pregnancy, infancy, toddler); and 3) total number of visits during each of the three program phases. The HAs enter all the data from the NFP nurse assessment forms into the provincial database systems. This information is utilized by the NFP PHNs and supervisors to guide their practice, guide program implementation, inform clinical supervision, enhance program quality, and assess program fidelity. The NFP Program Fidelity Reports are generated automatically from information recorded in the Home Visit/Alternate Encounter form, which is completed after each NFP home visit. The primary purpose of collecting these anonymized and aggregate fidelity data will be to provide a descriptive overview of the fidelity for some of the NFP model elements across local health areas and HAs.

Data analysis

Qualitative data from the interviews will be analyzed using directed content and thematic analysis, and constant comparative approaches [20]. The analytic process in interpretive description includes broad labeling of concepts through open coding, a synthesis of meanings across codes, the theorizing of relationships through the use of constant comparative methods and the contextualization of data into findings [20]. All interview data will be recorded and transcribed verbatim with any identifying information removed. Transcripts will be reviewed and cleaned by the research team administrative assistant and verified by the Research Coordinator. All textual data (field notes, transcripts, documents) will be imported to NVivo 10.0, a qualitative software package used for overall data management, including coding, searching and indexing.

Both directed (deductive) and conventional (inductive) content analytic procedures will be used to develop the codebook [21]. A set of preliminary codes will be derived from the research questions in the interview guides. However, new codes will be developed and defined as concepts are identified within the data. The codebook will contain the following elements: code name, full definition, instructions on when to apply the code, instructions on when not to apply the code and a brief example of data relevant to that code [22]. The initial code list and code definitions will be circulated to the members of the research team for review. Members of the research team will also independently code a sample of interview data and consistency of code application will be assessed. Based on these findings, the codebook will be reviewed and revised. The Research Coordinator will be subsequently responsible for applying the approved codes to the data and completing all first- and second-level coding.

Quantitative data from NFP Program Fidelity forms and information from the Case Conference forms will be collected and analyzed. Depending on sample size, results will be analyzed and reported at both the local health area and HA levels. The main quantitative analysis will consist of descriptive statistics designed to determine various aspects of program fidelity following recommendations by Boller and colleagues [23]. These fidelity indicators will be evaluated under five broad categories: 1) PHN and supervisor caseloads; 2) duration of the program; 3) service dosage of the program; and 4) content of home visits. According to NFP model fidelity requirements (adapted for Canada) a full-time PHN should carry a caseload of no more than 20 active clients and a full-time supervisor should provide supervision to no more than eight individual PHNs. Measures for this category will include mean monthly PHN and supervisor caseloads and the percentage of PHNs and supervisors at or below required caseload for the observation period. Nurse supervisors provide PHNs with clinical supervision through various activities including one-to-one clinical supervision, case conferences team meetings and field supervision. From the Case Conference forms, number of hours spent in 1:1 supervision (in person or teleconference) per month will be averaged and reported. Time spent in team meetings and conferences will also be averaged and reported. Finally, percentage of time spent on various themes during team meetings will be described.

The NFP seeks to enroll clients early in pregnancy and retain them until the child reaches their second birthday (typically 2.5 years duration). The percent of clients enrolled by 12-, 16-, and after 20 weeks of pregnancy will be described. Duration measures will include: percentage of participants with at least one home visit who remain enrolled at 3, 6, 12, 18 and 24 months; the percentage of participants who do not graduate from the program (at the child’s second birthday); and the mean duration of time spent in the program for those who leave (calculated from the discharge date).

Regular contact with families is one of the key facets of the NFP. NFP guidelines articulate expectations for program dosage varying over the stages of program delivery – pregnancy, infancy and toddlerhood. To assess dosage of the program, we will calculate the percent of participants receiving the intended dosage across all the three stages (90 and 80 % of intended dosage levels) and the number of visits provided between enrollment and date of exit. To fulfill our research objectives, qualitative interview and document data and quantitative NFP Program Fidelity data will be collected in parallel, analyzed separately, and then merged at the interpretation stage.

Rigor

To ensure the overall quality and trustworthiness of the process evaluation findings, a range of strategies will be applied across all phases of the study to achieve overall data credibility (internal validity), dependability (reliability or the consistency of the findings) and confirmability (neutrality) [24]. A summary of these strategies is provided in Table 5.

Table 5 Strategies to achieve rigor across the process evaluation

Ethical considerations

As the overarching BCHCP is a mixed methods study that includes the RCT and the process evaluation, both studies were submitted as a single application to ten Research Ethics Boards for review. Approval to conduct the BCHCP (inclusive of the process evaluation) was successfully obtained from all ten of these boards – which consisted of five boards from the participating HAs, four boards based in Universities where investigators held their affiliations or faculty appointments, and through the Health Canada and Public Health Agency of Canada Research Ethics Board. The names of the Research Ethics Boards that approved this study are listed in Table 6. Specific to the process evaluation component, informed consent to participate will be sought from all participants. All data will remain confidential and identifying information will be removed from all transcript data. As part of the consent procedure, PHNs and supervisors specifically will be informed about the potential for experiencing distress associated with participation in the process evaluation. The potential to experience distress may occur: 1) when talking to client participants and their families regarding their life circumstances; 2) during reflective supervision; or 3) when discussing their nursing education or role in the BCHCP. Participants will be given the opportunity to take a break or to stop the interview at any point.

Table 6 Names of Research Ethics Boards that reviewed and approved BCHCP process evaluation

The potential benefits of participating in the process evaluation will also be outlined for participants, including: 1) the experience gained through the NFP education and their involvement in the study is expected to facilitate substantial professional growth and development for participating PHNs and supervisors; and 2) information from this study may be used to help improve programs and services for young, first-time BC (and Canadian) mothers and their children in the future. Findings from the Ontario pilot study [8] suggested that almost all participating PHNs and the local supervisor found the NFP experience to be highly enriching and rewarding. By participating, PHNs and supervisors may be making a lasting contribution to these program improvements.

Discussion

Historically, much emphasis has been placed on determining and measuring the overall effectiveness of public health interventions through the conduct of RCTs. However, given the complexity of many of these interventions, and the challenges inherent in implementing these programs in different contexts, it is important for researchers to extend their evaluations to include measures of both outcomes and process. In the BCHCP, our process evaluation of the implementation and delivery of the NFP will provide us with important information about fidelity, the quantity of the intervention delivered and received, and the overall quality of the program delivered by HAs. Most importantly, from the process evaluation we will be able to provide a detailed description of the social-political-cultural contexts that are unique to each of the five distinct HAs delivering NFP in BC. This type of data can be useful in interpreting findings from the trial and establishing how generalizable the results will be to other contexts [25]. In BCHCP sites participating in the basic process evaluation (i.e., not enrolling eligible women into the RCT), data from the process evaluation will be essential in providing insights about how PHNs and supervisors were required to adapt model elements and program content to meet the needs of their clients, often residing in smaller suburban, rural and remote areas. This information will be valuable in informing future adaptations to the NFP model for similar populations in Canada and in other international contexts.

This process evaluation has several key methodologic strengths. First, as a mixed methods study, the combination of quantitative and qualitative methods will result in findings that offer a more comprehensive understanding of how the NFP is implemented in BC; the integration of the contextual qualitative data will ensure that the findings can be utilized conceptually by health decision makers. It has been suggested that research that includes three key domains (process, content, and outcomes) will be more likely to result in policy change or evidence-informed decision making within the public health arena [26]. The BCHCP, including findings from both the RCT and the process evaluation described here, therefore has a high likelihood of influencing policy change, given its mixed methods approach addresses all three domains. Second, the triangulation of multiple data types from a range of sources (including PHNs, supervisors and managers) across the full geographic region where the NFP is being evaluated will provide results with a high level of credibility and trustworthiness. With information from the process evaluation, we will also be able to explain variances in implementation and outcomes (measured in the RCT) across and within the HAs. Another significant strength of the process evaluation, and the overall BCHCP, is the strong, collaborative partnership that has been developed between the BCHCP Scientific Team and the senior policy-makers within BC’s Ministries of Health and Child and Family Development. As with all aspects of the BCHCP to date, relevant BC Government and HA decision-makers have been and will continue to be engaged in providing feedback on the process evaluation sampling and data collection procedures and tools. As we move into the BCHCP RCT and process evaluation, there will continue to be ongoing dialogue and problem-solving using our established research-policy partnership governance structures. These existing relationships and structures thus result in opportunities to ensure that important information about implementation, model fidelity and program delivery is disseminated in a timely fashion to policy-makers, informing their decisions.

One limitation of the process evaluation is the lack of opportunity to collect qualitative data from the pregnant women and first-time mothers, as well as their partners and family members, who are enrolled in the BCHCP and receiving NFP. While data about the dose of the intervention received will be captured at an aggregate level, we will not be able to collect information about their experiences of engaging in the NFP home visitation program and their perceptions on how this intervention influences their capacity to parent.

Overall, the results of this process evaluation will be of immediate instrumental use to the policy-makers to inform ongoing implementation and delivery of the NFP. Beyond the context of BC, the findings from our study will contribute to the emerging body of evidence surrounding: 1) professional practice issues in the field of nurse home visitation, 2) best practices for adapting the NFP model elements to meet the needs of families living in smaller suburban, rural and remote communities; 3) a deeper understanding of how social and health issues such as mental health problems including substance misuse and IPV impact a young mother’s capacity to parent; and 4) strategies to support professionals from the primary care, child welfare and public health sectors to work collaboratively to meet the needs of children and families at risk or experiencing maltreatment. If the BCHCP RCT demonstrates the effectiveness of the NFP compared with existing services, for the first time in the Canadian context, findings from the process evaluation will be invaluable in informing wide-scale implementation efforts in Canada. For example, process evaluation findings related to PHNs’ and supervisors’ experiences of completing the NFP education components would inform the development of a Canadian-specific NFP model of education delivery. Valuable information pertaining to strategies for the recruitment and retention of both PHNs and clients will also be available. Finally, the findings will be of great interest and assistance with evaluating and implementing the NFP internationally.

Abbreviations

BC:

British Columbia

BCHCP:

British Columbia Healthy Connections Project

HA:

Health authority

IPV:

Intimate partner violence

NFP:

Nurse-Family Partnership

PHN:

Public health nurse

RCT:

Randomized controlled trial

US:

United States

References

  1. 1.

    Nurse-Family Partnership; http://www.nursefamilypartnership.org (2011). Accessed 9 May 2015.

  2. 2.

    Olds DL, Sadler L, Kitzman H. Programs for parents of infants and toddlers: Recent evidence from randomized trials. J Child Psychol Psy. 2007;48:355–91.

    Article  Google Scholar 

  3. 3.

    Olds DL, Kitzman H, Knudtson MS, Anson E, Smith JA, Cole R. Effect of home visiting by nurses on maternal and child mortality: results of a 2-decade follow-up of a randomized clinical trial. JAMA Pediatrics. 2014;168:800–6. doi:10.1001/jamapediatrics.2014.472.

    Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    MacMillan HL, Wathen CN, Barlow J, et al. Interventions to prevent child maltreatment and associated impairment. Lancet. 2009;373:250–66.

    Article  PubMed  Google Scholar 

  5. 5.

    Nurse-Family Partnership: Nurse-Family Partnership model elements. http://www.nursefamilypartnership.org/communities/model-elements (2011). Accessed 9 May 2015.

  6. 6.

    Prevention Research Center for Family and Child Health: Nurse-Family Partnership international program. http://www.ucdenver.edu/academics/colleges/medicalschool/departments/pediatrics/research/programs/prc/research/international/Pages/international.aspx (2015). Accessed 9 May 2015.

  7. 7.

    Barnes J. From evidence-base to practice: implementation of the Nurse Family Partnership programme in England. J Child Serv. 2010;5:4–17.

    Article  Google Scholar 

  8. 8.

    Jack SM, Busser LD, Sheehan D, Gonzalez A, Zwygers EJ, MacMillan HL. Adaptation and implementation of the Nurse-Family Partnership in Canada. Can J Public Health. 2012;103 Suppl 1:S42–8.

    Google Scholar 

  9. 9.

    Kurtz Landy C, Jack SM, Wahoush O, Sheehan D, MacMillan L and NFP Hamilton Research Team. Mothers experiences in the Nurse-Family Partnership program: a qualitative case study. BMC Nurs. 2012;11:15.

    Article  Google Scholar 

  10. 10.

    Jack SM, MacMillan HL. Adaptation and evaluation of the Nurse-Family Partnership in Canada. Early Childhood Matters 2014, June: http://www.bernardvanleer.org/Responsive-parenting-a-strategy-to-prevent-violence. Accessed 9 May 2015.

  11. 11.

    Children’s Health Policy Centre: BC Healthy Connections Project. http://childhealthpolicy.ca/bc-healthy-connections-project/ (2015). Accessed 9 May 2015.

  12. 12.

    Medical Research Council. Developing and evaluating complex interventions: new guidance. London: Medical Research Council; 2008.

    Google Scholar 

  13. 13.

    Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496.

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Linnan L, Steckler A. Process evaluation for public health interventions ad research: an overview. In: Steckler A, Linnan L, editors. Process Evaluation for Public Health Interventions and Research. San Francisco: Jossey-Bass; 2002. p. 1–24.

    Google Scholar 

  15. 15.

    Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6:134–47.

    Article  PubMed  Google Scholar 

  16. 16.

    Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 2nd ed. Thousand Oaks CA: Sage Publications; 2011.

    Google Scholar 

  17. 17.

    Thorne S. Interpretive Description. Walnut Creek: CA: Left Coast Press; 2008.

    Google Scholar 

  18. 18.

    MacDougall C, Baum F. The devil’s advocate: a strategy to avoid groupthink and stimulate discussion in focus groups. Qual Health Res. 1997;7:532–41.

    Article  Google Scholar 

  19. 19.

    Jack SM, Ford-Gilboe M, Wathen CN, Davidov DD, McNaughton DB, Coben JH, et al. Development of a nurse home visitation intervention for intimate partner violence. BMC Health Services Res. 2012;12:50.

    Article  Google Scholar 

  20. 20.

    Thorne S, Reimer Kirkham S, O’Flynn-Magee K. The analytic challenge in interpretive description. Int J Qual Methods. 2004;3:1–11.

    Google Scholar 

  21. 21.

    Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88.

    Article  PubMed  Google Scholar 

  22. 22.

    MacQueen KM, McLellan E, Kay K, Milstein B. Codebook development for team-based qualitative analysis. Cultural Anthropol Methods. 1998;10:31–6.

    Google Scholar 

  23. 23.

    Boller K, Daro D, Del Grosso P, Cole R, Paulsell D, Hart B, et al. Making replication work: building infrastructure to implement, scale-up, and sustain evidence-based early childhood home visiting programs with fidelity, Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services. Princeton, NJ: Mathematica Policy Research; 2014. Contract No.: GS-10F-0050L/ HHSP233201200516G.

    Google Scholar 

  24. 24.

    Krefting L. Rigor in qualitative research: the assessment of trustworthiness. Am J of Occup Ther. 1991;24:214–22.

    Article  Google Scholar 

  25. 25.

    Wells M, Williams B, Treweek S, Coyle J, Taylor J. Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials. 2012;13:95.

    Article  PubMed  PubMed Central  Google Scholar 

  26. 26.

    Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99:1576–83.

    Article  PubMed  PubMed Central  Google Scholar 

  27. 27.

    Bronfenbrenner U. The Ecology of Human Development: Experiments By Nature and Design. Cambridge, MA: Harvard University Press; 1979.

    Google Scholar 

  28. 28.

    Bowlby J. Attachment and Loss. London, UK: Hogarth Press; 1969.

    Google Scholar 

  29. 29.

    Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psych Rev. 1977;84:191–215.

    CAS  Article  Google Scholar 

  30. 30.

    Prochaska JO, DiClemente CC. Toward a comprehensive model of change. In: Miller WR, Heather N, editors. Treating Addictive Behaviors: Processes of Change. New York: Plenum Press; 1986. p. 3–27.

    Google Scholar 

  31. 31.

    O’Brien R. Nurse-Family Partnership’s theory of change logic model http://www.nursefamilypartnership.org/assets/PDF/Communities/TOC-Logic-Model (2008). Accessed 9 May 2015.

Download references

Acknowledgements

Members of the BCHCP Process Evaluation Team include: Kaitlyn Hougham MSc (Simon Fraser University), Wendy Hovdestad PhD (Public Health Agency of Canada), Christine Kurtz Landy RN PhD (York University), Karen MacKinnon RN PhD (University of Victoria), Lenora Marcellus RN PhD (University of Victoria), Lil Tonmyr PhD (Public Health Agency of Canada), Natasha Van Borek MScPPH (McMaster), and Colleen Varcoe RN PhD (University of British Columbia). The overarching BCHCP is funded by the BC Ministry of Health with support from the BC Ministry of Children and Family Development and from Fraser Health, Interior Health, Island Health, Northern Health and Vancouver Coastal Health. Additional funding for the BCHCP process evaluation was received from the Injury and Child Maltreatment Section, Centre for Chronic Disease Prevention, Public Health Agency of Canada. Dr. Nicole Catherine is supported by the Mowafaghian Foundation. Dr. Harriet MacMillan is supported by the Chedoke Health Chair in Psychiatry. Dr. Charlotte Waddell is supported by the Canada Research Chairs Program.

Author information

Affiliations

Authors

Consortia

Corresponding author

Correspondence to Susan M. Jack.

Additional information

Competing interests

DS and SJ hold separate contracts with the Prevention Research Center for Family and Child Health, University of Colorado at Denver (DS-to provide consultation services to the NFP international societies; SJ-to consult on the piloting and evaluation of the NFP IPV nurse education, State of Colorado). All other authors declare that they have no competing interests.

Authors’ contributions

All named authors (SJ, DS, AG, HM, NC, CW) contributed to the conception and design of the BCHCP process evaluation. SJ drafted the manuscript and AG drafted the quantitative section. All authors read, provided input and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jack, S.M., Sheehan, D., Gonzalez, A. et al. British Columbia Healthy Connections Project process evaluation: a mixed methods protocol to describe the implementation and delivery of the Nurse-Family Partnership in Canada. BMC Nurs 14, 47 (2015). https://doi.org/10.1186/s12912-015-0097-3

Download citation

Keywords

  • Process evaluation
  • Home visitation
  • Public health nurses
  • Nurse-Family Partnership
  • Mixed methods
  • Intervention