A Spanish adaptation of the Quality in Psychiatric Care—Inpatient (QPC-IP) instrument: Psychometric properties and factor structure

Abstract

Background and aim

Western countries share an interest in evaluating and improving quality of care in the healthcare field. The aim was to develop and examine the psychometric properties and factor structure of the Spanish version of the Quality in Psychiatric Care–Inpatient (QPC-IP) instrument.

Methods

A psychometric study was conducted, translating the QPC-IPS instrument into Spanish, revision of the instrument by a panel of experts, and assessing its psychometric properties. 150 psychiatric inpatients completed the QPC-IP. Test-retest reliability was assessed by re-administering the questionnaire to 75 of these patients.

Results

After conducting pilot testing and a cognitive interview with 30 inpatients, it was determined that the QPC-IPS was adequate and could be self-administered. A Cronbach’s alpha of 0.94 was obtained for the full instrument and values of 0.52–0.89 for the various dimensions of the questionnaire. Test re test reliability: The Intraclass Correlation Coefficient for the full questionnaire was 0.69, while for the individual dimensions values between 0.62 and 0.74 were obtained, indicating acceptable temporal stability. Convergent validity was analysed using 10-point numerical satisfaction scale, giving a positive correlation (0.49). Confirmatory factor analysis revealed six factors consistent with the original scale. The Spanish version yielded adequate results in terms of validity and reliability.

Conclusion

Our findings provide evidence of the convergent validity, reliability, temporal stability and construct validity of the Spanish QPC-IP for measuring patient quality in psychiatric care in Spanish hospitals. Hospital administrators can use this tool to assess and identify areas for improvement to enhance quality in psychiatric care.

Background

Recent decades have seen a growing interest worldwide in measuring the quality of healthcare, not least in relation to mental health services [1]. However, these developments have not taken place to the same extent in all countries, nor even in different regions of the same country. One reasons for this is the lack of standardized measurement instruments for identify areas in need of improvement and for making national and international comparisons [2]. It has been observed in a recent systematic review [3] that quality instruments in mental health have psychometric properties with highly variable results and it is recommended to take into account those with high quality standards of results. For example, in the area of mental health the Quality of Psychiatric Care–Inpatient (QPC-IP) instrument [4] was developed with the aim of reduce this gap. The QPC-IP is part of a family of instruments that comprise both a common core and context specific: the QPC-OP for the psychiatric outpatient care [5], the QPC-FIP for psychiatric forensic inpatient care [6], the QPC-DA for daily activities in community-based services for people with psychiatric disabilities [7], and the QPC-H for the quality of community housing support [8]. Each QPC instrument also has a staff version, of which the one for psychiatric forensic inpatient care (QPC-FIPS) [9] has currently been validated. Adaptations of the QPC-FIP have been carried out in Denmark [10] and the QPC-IP and QPC-IPS in Indonesia [11, 12].

In Spain, legislation on the quality of public healthcare (Law 16/2003) has led to the creation of regional bodies with responsibility for assessing the quality of health services, especially mental health. In Catalonia, the Agency for Health Quality and Assessment (AQuAS) fulfils this role using validated satisfaction instruments. However, the concept of patient satisfaction does not encompass all aspects of quality of care. Questionnaires of satisfaction often involve questions that reflect the concerns of managers, such as health outcomes, rather than focusing on aspects which patients themselves might see as important [13]. Indeed, patient satisfaction is not necessarily synonymous with quality from the patient’s perspective [14], since measuring satisfaction in terms of service indicators is not the same as exploring patients definition of quality of care and what enable their recovery [15]. The essential components that make up this quality are the therapeutic setting, the therapeutic relationship and support, assessment, professional performance, assessment of practice, and environmental health [16].

Faced with this problem, the QPC-IP has the potential be a useful tool for assessing the quality of mental health services in Spanish-speaking countries. However, the QPC-IP was developed in the Swedish context, and according to the 6-D model of national culture [17], Spain and Sweden are culturally distinct, especially on the dimensions of uncertainty avoidance and masculinity-femininity, although they are similar in terms of power distance, individualism-collectivism, and long-term orientation. Given these cultural differences, a rigorous adaptation of the instrument is required, including analysis of its psychometric properties.

This research is part of an extensive study designed to adapt the instrument to other international contexts, analyze the psychometric properties and dimensions of the different instruments of the QPC-IP, and detail and compare the quality of hospital mental health care in the different territories.

Methods

The adaptation process and the psychometric evaluation was carried out in the context of two psychiatric services in Catalonia, one in the city of Barcelona and the other in the nearby town of Sant Boi de Llobregat. Meetings were held with staff to inform them about the purpose and nature of the study. Nurses and occupational therapists recruited patients who were willing to participate and who met the inclusion criteria. After obtaining informed consent, the staff instructed the patients on how to complete the questionnaire. Patients were included consecutively until the required sample size was reached.

Adaptation of Spanish inpatient instrument

Figure 1 shows the process of translation and back-translation of the development of the Spanish QPC-IP. The original (Swedish) version of the instrument was first translated into Spanish. The research team, comprising professionals from the fields of nursing and psychiatry, as well as health and care quality managers, reviewed the translation and checked that the meaning of each item was expressed and translated correctly (cultural validation). Each item was rated on a scale from 1 to 4 (minimum-maximum) with regard to its coherence, clarity, and relevance. This preliminary Spanish version was then back-translated into Swedish and sent to the Swedish research group. The Swedish research group assessed the degree of convergence between the back-translation and the original version. Following discussion and subsequent agreement regarding the semantic equivalence (face validity) of the Spanish QPC-IP the instrument was piloted in cognitive interviews with 30 psychiatric inpatients. This process confirmed that the Spanish version of the instrument was easy to understand and to answer.

Fig. 1

Overview of the three-phase validation study

Analysis of psychometric properties

Sample and participants

The required sample size was estimated in accordance with the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN), the Standards for Educational and Psychological Testing and the judgment of experts [18]. It was considered that a minimum of 5 patients per questionnaire item would be needed to assess internal consistency. For the analysis of temporal stability, we estimated that a minimum of 75 patients would be required and that the intraclass correlation coefficient between the two administrations should be ≥.70 [19], considering a 95% confidence level and power of 80% in a two-sided test [20].

The total sample therefore comprised 150 inpatients from different units of the two psychiatric services (acute, sub-acute, long-stay, therapeutic community as an open door unit with a three-month average stay, residential care where patients live permanently). Data were collected between September and December 2017. The inclusion criteria were [1] being an inpatient in one of the aforementioned units at the time of the study, [2] aged 18 or over, [3] having a diagnosed mental disorder, and [4] voluntary participation. The exclusion criteria were [1] unable to understand or communicate in Spanish, [2] significant cognitive impairment, [3] currently in seclusion, [4] currently under mechanical and/or pharmacological restraint, [5] learning disability, [6] organic disorder and/or intoxication due to drug use.

Sample description

A total of 150 patients (45.3% female, 54.7% male) aged between 18 and 72 (mean = 43.6, SD = 12.63) completed the QPC-IP. The 86% of the sample were Spanish. Regarding marital status, 62% of admitted patients were single and 15.3% divorced or widowed. Regarding the education received, 26.6% completed primary school or did not finish them, and 30.7% of the population specialized in job training. In relation to the main occupation, 44% received the sickness pension. The average number of days of admission was 43 (SD 11.0). The 73.3% of the sample had been admitted in the hospital more than once. The average number of readmissions corresponded to 5.89 (SD 9.89). 65.3% of the total knew their diagnosis. According to the type of admission, 53.3% had a voluntary admission.

The instruments

The quality in psychiatric care-inpatients

The QPC-IP was developed by Schröder et al. [21] for use in a specific mental health context and with the aim to give the patients’ views a central role in the assessment of psychiatric care quality. QPC-IP is a 30-item self-administered instrument based on a definition of quality of care from the patient’s perspective [3] that was developed through a phenomenographic interview study with inpatients and outpatients. The items cover six dimensions of psychiatric care, labeled as follows: Encounter (8 items), Participation (8 items), Support (4 items), Discharge (4 items), Secluded Environment (3 items), and Secure Environment (3 items). The Cronbach’s alpha values of the six dimensions ranged between .75 and .95. Each of the 30 items of the QPC-IP is rated on a 4-point Likert-type scale ranging from 1 (totally disagree, indicative of lowest quality) to 4 (totally agree, indicative of highest quality), and thus the total score ranges from 30 to 120. The definition was developed from a phenomenographic interview study [22], and the instrument was tested for face validity in a pilot study and also empirically tested [4].

10-point numerical scale satisfaction

A numerical scale of satisfaction scores has been used, ranging from 0 to 10, with 0 being the lowest score and 10 the highest score being the best satisfaction score.

Data analysis

The reliability (internal consistency) of the Spanish version of the QPC-IP was tested by computing Cronbach’s alpha coefficients for the total questionnaire and for each of its six dimensions. A value above .70 was considered adequate [23]. To test for scale homogeneity we calculated corrected item-total correlation coefficients, that is, the correlation of each item with the total score and with the score on each dimension when that item is omitted, accepting a correlation of .30 as the lower limit [23].

Test-retest reliability (temporal stability) was assessed over an interval of 7–14 days, re-administering the Spanish QPC-IP to 75 of the total of 150 patients. For this analysis we calculated the intraclass correlation coefficient (range 0 to 1) and interpreted a value ≥.70 as indicating good agreement [19], considering a 95% confidence level and power of 80% in a two-sided test [20].

To analyze convergent validity we calculated the Spearman (rho) correlation coefficient between scores on the Spanish QPC-IP and a 10-point numerical satisfaction scale on which 0 and 10 corresponded to the lowest and highest satisfaction rating.

Construct validity was tested by means of confirmatory factor analysis (CFA), with parameter estimates being obtained using the generalized least squares method and the EQS. 6.1 software package. This method is identical to the maximum likelihood estimator, but it has less strict normality criteria and is mainly used with ordinal items. The overall fit of the model was determined by calculating both absolute and incremental fit following indices: the GFI (goodness-of-fit index), the AGFI (adjusted goodness-of-fit index), the RMSEA (Root mean square error of approximation), the TLI (Tucker-Lewis index for comparison), the CFI (Comparative fit index), the BBNFI (Bentler-Bonett normed fit index), and the BBNNFI (Bentler-Bonett non-normed fit index). The criteria for a good fit were values of the GFI, AGFI, TLI, CFI, BBNFI, BBNNFI >.90, and a value of the RMSEA <.08 [24, 25]. We also calculated the reduced chi-squared statistic, defined as the ratio of the chi-squared value to the number of degrees of freedom. Values between 2 and 6 were considered acceptable [26].

A 95% confidence level was used for all the aforementioned statistical tests. Descriptive statistics were analyzed using SPSS 22.

Ethical considerations

The study was approved by the Clinical Research Ethics Committee of our hospital (CEI PIC− 128 − 15) and permission was granted by the coordinators and supervisors of the respective psychiatric units. All questionnaires were confidential, and all the patients signed informed consent in accordance with existing Spanish legislation. Their participation was voluntary.

Results

Adaptation of Spanish inpatient instrument

For the adaptation of the Spanish instrument, a panel of experts produced the results based on coherence, clarity, and relevance greater than 3. No items required modification. After conducting a pilot testing and a cognitive interview with 30 inpatients, it was determined that the QPC-IPS was adequate and could be self-administered. The results of this phase were positive, and there were no problems in the comprehension or administration of the questionnaire.

Analysis of psychometric properties

Table 1 shows the descriptive statistics of items from SpanishQPC-IP questionnaire.

Table 1 Descriptive Statistics of items from SpanishQPC-IP questionnaire

Construct validity

In the confirmatory factor analysis (CFA) all factors loadings were statistically significant (Fig. 2). The highest loadings were found for the Participation (Factor 1) and the Discharge dimensions (Factor 6). The correlations between factors were high in all cases, with the exception of that between Secure environment (Factor 5) and Discharge (Factor 6) (value of .50).

Fig. 2

Factor loadings derived from the LS estimation (least squares) Confirmatory Factor Analysis (λij)

The result of the chi-square test was significant (χ² = 935.500; p < .001), suggesting inadequate model fit. However, since the chi square test is sensitive to sample size we proceeded to calculate the ratio between chi-square and the degrees of freedom (χ² / df). The value obtained was 2.39, wish is well within the 2 and 6 range considered to indicate acceptable fit [27]. As the results of our initial CFA indicated a positive latent variable covariance matrix. The results obtained ranged from adequate to excellent. Furthermore, the other fit indices confirmed an adequate fit of the model (Table 2).

Table 2 Goodness of fit indices of Model Confirmatory

Convergent validity

Analysis of the correlation between scores on the Spanish QPC-IP and a 10-point Numerical Scale Satisfaction yielded a Spearman’s rho coefficient of .49. The correlation of total and each subdimension of the QPC-IP was 0.86 (F1: Encounter), 0.89 (F2: Participation), 0.74 (F3: Support), 0.67 (F4: Seclude Environment), 0.63 (F5: Secure Environment), and 0.76 (F6: Discharge).

Internal consistency

As shown in Table 3, the Cronbach’s alpha for the total scale was .94, and five of the six dimensions yielded a value above .70. The exception was ‘Secluded environment’, which received an alpha of .67. We also calculated alpha values for the total scale and each of its dimensions when excluding one item at a time. Internal consistency was not notably improved by excluding any of the items.

Table 3 Coefficient of internal consistency of SPANISH-QPCIP

Test-retest reliability

Temporal stability was assessed by 75 drawn from the total of 150 patients. The intraclass correlation coefficient for the total scale was .69 (95% CI: .52–.81), ranging between .62 and .74 for the six dimensions (Table 4).

Table 4 Intraclass Correlation Coefficient (ICC) test-retest

Descriptions of quality of inpatient care

As shown in Fig. 3, the perception of quality based on the Encounter dimension was significantly higher than the second factor, Support (t₍₁₄₉₎ = 3.18, p = 0.04), which was perceived as higher than Discharge (t₍₁₄₉₎ = 3.04, p = 0.00). The remaining three dimensions were rated from highest to lowest; the Participation being greater than Secure Environment (t₍₁₄₉₎ = 2.89, p = 0.23), and greater than Secluded Environment (t₍₁₄₉₎ = 2.63, p = 0.00).

Fig. 3

Mean ratings of the Spanish version of the QPC-IP dimensions. Error bars represent 95% Confidence interval

Discussion

The purpose of this study was to develop and examine the psychometric properties and factor structure of a Spanish adaptation of the QPC-IP instrument. The results show that the Spanish QPC-IP has adequate psychometric properties in terms of internal consistency, temporal stability (test-retest), content and construct validity (confirmatory factor analysis) and convergent validity. The CFA revealed an adequate fit of the six-factor structure consistent with the original Swedish version.

It was determined that the instrument was adequate and could be self-administered after conducting a pilot test and a cognitive interview with 30 patients. The results of this phase were positive and there were no problems in the comprehension or administration of the questionnaire. Although 13.3% of the sample did not complete comprehensive school, no support was needed for the patients to complete the instrument.

Internal consistency was acceptable (α ≥ .70) for the total scale and for five of the six dimensions. The highest alpha value was found for the Encounter dimension and the lowest for Secluded environment dimension (α = .67). Since Cronbach alpha is highly influence by the number of items in a dimension, the low alpha value of Secluded environment is probably due to the fact that it only include three items. However, the internal consistency of Secluded environment in other language versions of the QPC-IP, such as the Indonesian one [12], also shows a low Cronbach alpha value, suggesting that this dimension might be more cultural specific than the other dimensions in the QPC-IP. Overall, the magnitude of the alpha values for individual items showed a reasonable degree of variation, suggesting that none of the items was perceived as confusing or upsetting. It should be noted that the alpha values are adequate according to established criteria [23] and are higher than those reported for the original version [21] and in other studies that have used the OPC-IP [11]. It is also worth pointing out that the reliability results obtained for the Spanish QPC-IP are similar to other quality of care measures reported in a recent systematic review [3]. Specifically, instruments yielding similar results to those of the Spanish QPC-IP are the SEQUenCE (Service user Quality of CarE) instrument [28], with a Pearson correlation coefficient of .65 and a Cronbach’s alpha of .87, the MQOC (Menninger Quality of Care) measure [29], with an alpha of .92, and the QPC-IPS with an alpha 0.92 [30].

Regarding test-retest reliability, the value of the intraclass correlation coefficient obtained in the present study (ICC = .69) was acceptable. The Discharge dimension (ICC = .681) showed the lowest ICC values. One of the reasons for the low ICC values may be that patients discharge from ward were not scheduled in advance.

The convergent validity of the Spanish QPC-IP was examined by calculating the Spearman rho correlation coefficient with respect to a 10-point numerical satisfaction scale. The result was interpreted according to the criteria proposed by Martínez González, Sánchez Villegas, Toledo Atucha, and Faulin-Fajardo [31], namely: a value of zero, no correlation; <.30, weak association, ≥.31 ≤ .70, moderate association; >.71, strong association. The value obtained (rho = .49, p < .001) indicates a moderate association, showing that the higher the level of satisfaction the more positive is the patients’ rating of quality of care.

Regarding construct validity, the CFA yielded a six-factor model consistent with the original QPC-IP [21], thus confirming the multidimensional nature of the concept of quality [32]. All the item loadings were above .30, which is considered an acceptable minimum [33]. The goodness-of-fit indices also indicated a reasonably good fit of the model [25]. Overall, the results of the CFA support the validity of the Spanish QPC-IP and are in line with those reported for the QPC-IP [21], the QPC-OP [5] the QPC-FIPS [32] the Danish QPC-FIPS [34], and the Indonesian QPC-IP [12] and the Indonesian QPC-IPS [11].

It should be noted, however, that some differences were observed with respect to the recently validated Indonesian adaptation of the QPC-IP [12], possibly due to cultural differences with respect to our country.

However, given the present results, we can see that the patients in this study rated the highest quality in the Encounter dimension, which is in the line with previous studies on inpatient psychiatric care [12, 21]. The Secluded Environment was rated the lowest in terms of quality of care whereas in the study by Schröder et al. [21] performed in Sweden the Secluded Environment dimension was the second highest in terms of quality. One possible explanation may be that in Spanish psychiatric units there are no individual rooms available, except for isolation rooms for emergency crisis. However, it should be noted that the Secluded environment dimension in the Spanish QPC-IP fitted the original Swedish model whereas in the study by Lunqvist et al., [12], performed in Indonesia, the Secluded Environment dimension did not fit the model of the original scale. This result indicates that the concept of secluded environment is fairly similar in Spain and Sweden but distinctly different between Indonesia and Sweden, and possibly Spain too.

The extensive validation process in adapting the QPC-IP to the Spanish situation is a strength of the present study. However, a number of limitations also need to be acknowledged. First, although we estimated the minimum sample size needed for the present analysis, further studies with larger samples are required to confirm the factor structure obtained here. Second, the instrument has been developed with the specific purpose of assessing quality of care from the psychiatric inpatients’ perspective. Further studies are therefore needed to adapt the QPC instrument for use with Spanish-speaking populations in other contexts, such as outpatients. It would also be of interest to analyze the predictive capacity (sensitivity and specificity) of the Spanish QPC-IP instrument on patient recovery.

Conclusion

The Spanish QPC-IP is a simple and easily administered tool for measuring various aspects of quality in psychiatric inpatient care from the patients’ perspective. The six-factor structure and psychometric property of the Spanish QPC-IP are consistent with those of the original instrument, supporting its use as a measure of quality of care in Spanish-speaking populations. In this respect, it has the potential be used in cross-cultural comparative studies of quality of care in mental health. The results of these studies can be used to improve the quality of the provided service. Future studies will need to look at the psychometric properties of this instrument in relation to other variables and other samples of patients, both in the community and in other settings.