Impact of random safety analyses on structure, process and outcome indicators: multicentre study

Background

To assess the impact of a real-time random safety tool on structure, process and outcome indicators.

Methods

Prospective study conducted over a period of 12 months in two adult patient intensive care units. Safety rounds were conducted three days a week ascertaining 37 safety measures (grouped into 10 blocks). In each round, 50% of the patients and 50% of the measures were randomized. The impact of this safety tool was analysed on indicators of structure (safety culture, healthcare protocols), process (improvement proportion related to tool application, IPR) and outcome (mortality, average stay, rate of catheter-related bacteraemias and rate of ventilator-associated pneumonia, VAP).

Results

A total of 1214 patient-days were analysed. Structure indicators: the use of the safety tool was associated with an increase in the safety climate and the creation/modification of healthcare protocols (sedation/analgesia and weaning). Process indicators: Twelve of the 37 measures had an IPR > 10%; six showed a progressive decrease in the IPR over the study period. Nursing workloads and patient severity on the day of analysis were independently associated with a higher IPR in half of the blocks of variables. Outcome indicators: A significant decrease in the rate of VAP was observed.

Conclusions

The real-time random safety tool improved the care process and adherence to clinical practice guidelines and was associated with an improvement in structure, process and outcome indicators.

The application of evidence-based medicine is of major concern in intensive care medicine today [1]. Errors in health care may occur due to an unintended act or by omission. Those resulting from the former are more visible and therefore more easily detectable. Errors of omission are more insidious and more difficult to identify and include the failure to ensure that patients receive recommended medical care as supported by high-quality clinical research evidence [2], which occurs paradoxically in more severe patients [3]. For example, the lack of adherence to clinical practice guidelines may be due to the lack of knowledge about them and the presence of barriers that prevent their use such as a lack of time, a lack of resources, organizational aspects or even resistance to changing work habits.

To analyse and prevent patient safety-related incidents, reactive or proactive tools are used. These are complementary to each other. Checklists have been proposed as a simple and useful proactive method to prevent errors of commission and omission in critically ill patients [4, 5]. The complex reality in which they need to be implemented requires an approach that includes more than eliminating barriers and supporting facilitating factors. Implementation leaders must facilitate team learning to foster the mutual understanding of perspectives and motivations and the realignment of routines [6].

Among the various proactive methods, random safety audits [7] facilitate the interaction between the responsible team and the professional who verifies the safety measures, and have the potential to reduce future errors through the identification of system failures that contribute to gaps in quality and safety. This tool promotes the changes in accordance with the application of scientific evidence, feedback with the team, and providing and strengthening knowledge [8]. Weiss et al. [9] showed that checklists of safety measures guided by an observer (prompter) decreased mortality and average length of stay in an intensive care unit (ICU) compared to those carried out through self-verification.

Our group has developed and validated a tool—the real-time random safety audits (in Spanish: Análisis Aleatorios de Seguridad en Tiempo Real, AASTRE)—and found it to be effective in detecting and remedying errors of omission in real time, thereby improving adherence to guidelines [10] and proving to be most useful in situations of high care load and in more severe patients [11].

Thus, this multicentre study aims to investigate the usefulness of the AASTRE by measuring their effect on structure, process and outcome indicators.

Study design and participating centres

This is a prospective study involving two university hospitals over a 1-year period (January–December 2013). Table 1 shows the characteristics of the two centres and the most relevant initiatives implemented in terms of patient safety.

Methodology for the implementation of the AASTRE

Design and description of the checklist

The checklist, as previously validated [10], consists of 37 safety measures grouped into ten blocks of different areas of care: mechanical ventilation, haemodynamics, renal function and continuous renal replacement techniques (CRRT), sedation and analgesia, treatment (two blocks), nutrition, techniques and tests, nursing care and structure. AASTRE are standardly performed three days per week (including weeks with weekday holidays and holiday periods), with 50% of the safety measures and 50% of the ICU patients randomized on each day of analysis. Each safety measure has a specific definition, assessment criteria and a specific methodology for verification. All patients admitted to the ICU are eligible for AASTRE to be performed on them. However, for each selected patient, only those measures for which they meet the assessment criteria will be evaluated [11].

Role and training of prompters

The safety audits are always carried out immediately after the ICU daily clinical round and require the participation of a prompter and the healthcare professionals directly responsible for patient care (senior attending physician, residents and nurses). The prompter is one of the two senior attending physicians of each ICU (not directly caring for the patient) who has received the education and training required by the study and who is responsible for verifying and/or promoting the safety measures. At all centres, training sessions were held on the theoretical aspects and the methodology used in the AASTRE. In addition, all prompters were trained online in the goals of the study and in the use of the tool. Moreover, practical training was also required, carrying out at least three safety rounds prior to the start of the study.

Safety audits

Many of the measures included in the checklist are routinely carried out by healthcare professionals during the ICU daily clinical round. The purpose of the safety audits is to verify that they have indeed been carried out. If this were not the case (error of omission), the prompter reminds the healthcare professionals that they should be carried out. In this framework, the possible responses during the audits are: (1) “Yes”—when the measure analysed had been taken/performed on the ICU daily round; (2) “Yes, after AASTRE”—when the safety audit was used to detect an error of omission that has been corrected; (3) “No”—when the measure analysed could not be changed despite the audit; (4) “Not applicable”—when the patient did not meet the assessment criteria. The checklist and the responses of the evaluations are entered into a web platform (http://www.aastre.es). Safety audits were performed with a tablet at the bedside to facilitate implementation.

Definition of variables and indicators

1. 1.

   Number of patient-days was the number of patients assessed in the total number of days on which safety audits were carried out in the two hospitals.

2. 2.

   Structure indicators

   • Perception of safety culture (in Hospital 2): We used a previously validated questionnaire [12] based on the Safety Climate Survey (SCS) and the Safety Attitude Questionnaire-ICU model (SAQ-ICU). It analysed six dimensions: teamwork climate, safety climate, perceptions of management, job satisfaction, working conditions, and stress recognition. The questionnaire on the perception of safety culture was administered to medical, nursing and ancillary staff. Three evaluation periods were considered: 1) initial period: the month prior to the start of the study; 2) intermediate period: month 6 of the study; 3) final period: the month after the end of the study.

   • The execution or updating of protocols and/or procedures promoted by the AASTRE was investigated.

3. 3.

   Process indicators

   • The proportion of changes in the care process carried out as a result of verification was considered. IPR-AASTRE (improvement proportion related to the AASTRE) were calculated globally (IPR-AASTRE-G), for each safety measure (IPR-AASTRE), and for each block of variables (IPR-AASTRE-B), according to the following formulas:

     $$ {\text{IPR}} - {\text{AASTRE}} = \frac{{{\text{number of occasions on which the AASTRE changed clinical practice }}\left( {\text{{"yes}}, {\text{after the AASTRE"}}} \right)}}{{{\text{number of occasions on which the measure was selected }} - {\text{number of occasions on which the measure was not applicable}}}} \times 100 $$
     $$ {\text{IPR}} - {\text{AASTRE}} - {\text{B}} = \frac{\text{sum of the number of occasions on which the AASTRE changed clinical practice in each block}}{{{\text{number of occasions on which the measure was selected in each block}} - {\text{number of occasions on which the measure was not applicable in each block}}}} \times 100 $$

     IPR-AASTRE-B helped simplify the assessment of the impact of other variables on utility. These variables are: type of patient (medical, surgical, neurocritical and trauma), staffing ratio [PNR; patient:nurse ratio (≤2:1 vs. >2:1) and PPR; patient:physician ratio (≤2:1, 2–3:1, >3:1)], the Sequential Organ Failure Assessment (SOFA) score and length of stay (length of stay at the time of safety audits (<7, 7–14, >14 days).

1. 4.

   Outcome indicators

   • The impact by the AASTRE on ICU mortality, average stay and rates of central venous catheter-related bacteraemia (CRB) and ventilator-associated pneumonia (VAP) using standardized definitions [13, 14] was investigated. The clinical definition of VAP requires patients to fulfil one radiographic, one systemic, and two pulmonary criteria. Radiographic criteria include new or progressive infiltrates, consolidation and cavitation. Systemic criteria include fever, abnormal white blood cell count and altered mental status. Pulmonary criteria include purulent sputum, new or worsening cough or dyspnoea or tachypnea, rales or bronchial breath sounds, and worsening gas exchange. CRB is defined in a patient with a central venous catheter with at least one positive blood culture (two blood culture if common skin contaminant organism) obtained from a peripheral vein, clinical manifestations of infections (i.e. fever, chills and/or hypotension), and no apparent source for the bloodstream infection except the catheter. One of the following should be present: a positive semi-quantitative (>15 CFU per catheter segment) or quantitative (>10² CFU per catheter segment) catheter culture, whereby the same organism (species) is isolated from a catheter segment and a peripheral blood culture; simultaneous quantitative blood cultures with a ratio of >3:1 CFU/ml of blood (catheter vs. peripheral blood); differential time to positivity (growth in a culture of blood obtained through a catheter hub is detected by an automated blood culture system at least 2 h earlier than a culture of simultaneously drawn peripheral blood of equal volume). The information relative to VAP and CRB was collected prospectively at both centres participating in the study and the previous year, using identical diagnostic criteria.

Statistical analysis

For descriptive analysis, we used absolute numbers (N) and relative frequency (percentage) for categorical variables; the mean and standard deviation for continuous variables. Chi-square tests and linear trend Chi-square tests were used for categorical variables and Student’s t test for continuous variables in univariate analysis. For multivariate analysis, multiple logistic regression, fixed model and likelihood ratio method analyses were performed to ascertain the impact of different variables on the IPR-AASTRE-B and with the aim of adjusting for possible confounding effects. The results were expressed as odds ratio and their 95% confidence interval (CI).

We used direct standardization by APACHE 2012 (<15, 15–25, >25) to evaluate mortality change and incidence density ratio (IDR) 2013 vs 2012 and CI to evaluate CRB and VAP incidence changes. The acceptable level of statistical significance was set at p ≤ 0.05. All data analyses were performed using the SPSS version 15 statistical package (SPSS Inc., Chicago, IL).

During the study period, AASTRE were carried out on 1214 patient-days. Table 2 shows the distribution of the type of patients evaluated globally, and in each hospital, the workloads (of nursing staff and physicians), the seriousness of the patients measured using the SOFA and patients’ length of stay on the day the safety rounds were conducted. Most patients were medical (47.0%), with a PNR ≤2:1 (57.0%), a PPR 2–3:1(62.1%), SOFA <4 (56.9%) and an average stay <7 days (42.5%). It should be noted that the distribution of the types of patients evaluated is different in the two hospitals of the study. In Hospital 1, there was a predominance of medical patients (57.6%), followed by surgical patients (32.7%). In Hospital 2, although the evaluation of medical patients predominated (41.3%), followed by surgical patients (31.6%), there was a significantly higher percentage of assessments of neurosurgical (13.4%) and trauma patients (13.7%). The nursing workload was higher in Hospital 1, where in most cases each nurse takes care of more than two patients. With regard to the physicians’ workload, it was significantly higher in Hospital 1. It is in this centre that most frequently a physician treats more than three patients (36%). In terms of patient severity on the day of the administration of the AASTRE, the only differences found were in the SOFA subgroup <4 (more prevalent in Hospital 1, 67.1%) and in the SOFA subgroup 4–7 (more prevalent in Hospital 2, 32.7%). Finally, in respect of length of stay in the ICU on the day of the AASTRE, in Hospital 1 there was a significant predominance of patients whose length of stay was less than seven days (54%), the rest of the periods considered were significantly more prevalent in Hospital 2 except the period of ≥21 days, which was virtually identical in both hospitals.

Structure indicators

Perception of safety culture: The response rate to the perception of safety culture questionnaire that had been administered to 71 professionals was 94.4% (in the initial period), 66.6% (in the intermediate period) and 70.4% (in the final period). A progressive increase was observed in positive responses in the Safety Climate item throughout the study period (p < 0.0001) in the safety culture perception survey. No significant changes were observed in the other items (Table 3).

Implementation or updating of protocols and/or procedures: The use of the AASTRE was associated with changes in sedation/analgesia and weaning protocols at both hospitals. It is also noteworthy that in the two hospitals of the study, the use of AASTRE motivated the creation of a new procedure of the prescription and review of monitoring and mechanical ventilation (MV) alarms.

Process indicators

The overall IPR-AASTRE-G were 6.7%. Table 4 shows the distribution of patients evaluated for each measure and improvement proportion related to the AASTRE (IPR-AASTRE), and their evolution throughout the study period. Twelve of the 37 measures (32.4%) had IPR-AASTRE >10%. Some are included in the bundles to prevent VAP (evaluation of the level of sedation and pain in the sedated patient, semi-recumbent position) and CRB (daily assessment of catheter needs); others in the good medical practice guidelines (verification of alveolar pressure in patients with acute respiratory failure, assessment of acute renal failure and artificial nutrition) and in the basic safety measures (appropriate treatment prescription, review of MV or monitor alarms, patient identification).

Only six steps (verification of MV or monitor alarms, proper administration of the prescribed treatment, assessment of acute renal failure and the risk of developing pressure ulcers and prevention of thromboembolic disease) showed a progressive decrease in the IPR-AASTRE throughout the study period. In addition, in one measure (“daily assessment by parenteral nutrition team”), a significant increase was seen in IPR-AASTRE as it was assessed during the different four-month periods.

Table 5 shows the impact of the independent variables selected (type of patient, staffing ratio, severity and length of stay) in the IPR-AASTRE-B. The high PNR was associated with a higher IPR-AASTRE in the MV and haemodynamics blocks. The SOFA was associated independently with a higher IPR-AASTRE-B in four blocks. Finally, the length of stay was significantly inversely associated with the IPR-AASTRE-B of the techniques and tests and treatment blocks.

Outcome indicators

The use of the AASTRE was associated with a significant decline in the VAP rate. No significant impact on average stay, mortality and CRB rate was observed (Table 6).

Checklists have been proposed as tools to ensure that essential components of care are not omitted [15]. However, this is the first multicentre study to analyse the impact of real-time random safety audits on quality indicators in the critical patient. An improvement is seen in indicators of structure (safety climate, clinical protocols and healthcare procedures), process (better adherence to good clinical practice guidelines) and outcome (decline in the rate of VAP) [16]. These data support a way to improve health care for the critical patient by means of the AASTRE tool whose use is feasible as shown in the pilot study published previously by our group [10].

Structure indicators

The use of the AASTRE was associated with an improvement in the safety climate. An association has been described between a better safety climate and outcome [17], average stay [18] or adverse events [19]. Although other authors have not demonstrated that checklists improve communication and teamwork [20], the impact of the AASTRE on the safety climate could be the result of improved communication in clinical practice, as described in other tools [21].

The guidelines require local adaptation via local protocols to enable their effective, safe and efficient use [22, 23]. The AASTRE were associated with the need to renew sedation/analgesia and weaning protocols. This occurs as a natural consequence when verifying the safety measures using AASTRE reflexively and at the bedside. This highlights the need to update local protocols in accordance with the latest sources of scientific knowledge. Difficulties for adherence have been described [24, 25] in both protocols. The AASTRE allow evaluating adherence to protocols and can promote their regular updating. The AASTRE have improved safety in relation to monitoring and mechanical ventilation alarms through the creation of specific protocols.

While the beneficial effect of the introduction of protocols in clinical practice has been discussed [26, 27], most studies acknowledge that they are useful although more patently in hands of inexperienced healthcare providers or suboptimal work environments. In such malfunctioning environments, they help but may hinder the performance and progress of the health professional, reducing their autonomy [28]. In fact, in a study published recently, no effect of the protocols was observed in the outcome [29]. Therefore, AASTRE promote much needed professional autonomy but invites reflection as to decisions at the bedside. And this reflection leads to the updating of protocols even though a direct improvement in the results is not guaranteed.

Process indicators: IPR-AASTRE

Health care requires many more scientifically sound process measures than are currently available. The AASTRE are process indicators since they evaluate the degree of adherence to scientific evidence [30]. They allow measuring the gap between the indication of therapies that have proven effective within human clinical research and the real safe and effective use of these therapies in routine clinical practice.

The failure to ensure that patients receive recommended medical care is supported by high-quality clinical research evidence. This type of safety and quality problem can be effectively addressed with knowledge translation tools [23].

The fact that 12 of the 37 measures considered (32.4%) had IPR-AASTRE >10% shows the ability of the AASTRE to modify essential aspects of clinical practice and improve adherence to evidence-based guidelines, a priority in health care [11, 23, 31]. Hopefully, through organizational learning, this effect could be maintained over time [32]. In this regard, some authors [33] have described the ability of checklists to maintain adherence to good clinical practice guidelines achieving close to 100% compliance for semi-recumbent position or suitable sedation. However, in our study, these measures scored IPR-AASTRE of 21.7 and 11.6%, indicating that if the intervention (AASTRE) had not been implemented, the measure would not have been carried out in a large percentage of patients. Also, the evaluation of another essential measure as is the assessment of catheter needs, in our experience, was corrected in 16.8% of evaluations. The fact that IPR-AASTRE utility is maintained over time may be related to the complexity of ICU clinical activity. In this regard, the AASTRE act as a tool that redirects healthcare activity towards essential aspects of care, regardless of the environmental situation. However, in our study six measures showed a significant decrease along the four-month periods analysed, indicating that this tool can also help systematize healthcare and organizational learning. Nevertheless, in the case of measures with a gradually ascending IPR-AASTRE, it should be verified (as occurs in this study with the measure “daily assessment by parenteral nutrition team”) that the lack of adherence to the recommendations can be accounted for by causes from outside the work team implementing AASTRE (a problem of communication with other agents involved in treating the patient, as might occur in this case with professionals of the Pharmacy Service who are responsible for monitoring hospital parenteral nutrition, for example).

The AASTRE have proven to be more effective in more serious patients, in the early days of admission and in increased workload environments. These findings are consistent with the data published by our group previously [11]. Without interrupting the work flow, aspects of severe patient care are recalled and their definitive inclusion into treatment is left to the discretion of the senior physician responsible of the patient, according to the indication:risk ratio.

Outcome indicators

The concept of care bundling and its efficacy in improving clinical outcomes are also supported in the literature [34]. Our results show a significant decrease in VAP influencing three of the recommendations established to prevent this type of adverse event (assessment of sedation level, semi-recumbent position and prophylaxis of deep venous thrombosis). Dubose et al. [35] described this effect at a trauma ICU through a checklist of VAP bundle measures. However, in our study, the AASTRE had no impact on mortality and CRB. Probably, to demonstrate an impact on the rate of CRB requires influencing other aspects such as catheter insertion and maintenance [36].

In the critical patient, no study has managed to associate the use of safety checklists with a decrease in mortality [15]. Recently, in a study of Brazilian ICUs [37], the introduction of daily checklists, goal setting and clinician prompting did not decrease in-hospital mortality or other clinical outcomes. Despite being a study with a considerable sample size, some organizational and methodological aspects could render the results unreproducible outside that environment. For example, the health system is not comparable to the European one as regards cultural aspects and the organization of work teams. Moreover, standardized mortality is high and the number of patients recruited in each ICU was relatively low. In addition, important methodological aspects such as the period of analysis (just 4 months in that study), the definition of the measures, the eligibility of the patients and training in the use of the tools are different in the two studies. Nevertheless, the most distinguishing factor between the studies is the role of the prompter. According to the authors of the Brazilian study, the feedback of the clinician with the prompter was carried out later in the day. In our study, this is one of the keys of our methodology, the prompter interacts at the bedside during healthcare activity, immediately after the daily clinical rounds, acting as a catalyser of the transfer of knowledge, thus improving adherence to scientific proof. In any case, we are aware that a single intervention, albeit cross-cutting, never has a definitive impact on patient prognosis. Moreover, using mortality as an outcome measure requires larger samples and risk adjustment for fair comparison among providers and organizations [38].

There are limitations to this study. (1) Only two ICUs have participated. Moreover, their participation in the design of the AASTRE tool, the experience gained by the research team from the pilot study and the development of the culture of the continuous improvement of the quality of care that is underlying in the participating centres may mean that it is not possible to extrapolate the results to other ICUs. (2) The Hawthorne effect, a performance gain resulting from the knowledge of being observed, is difficult to distinguish from those resulting from the intervention. (3) The perception of safety culture was investigated only at one centre. (4) Sample size was not initially calculated to investigate the impact of the AASTRE on mortality or nosocomial infection rates. (5) The study design does not include a control group, since the random selection of the patients evaluated in the safety audits does not allow this. (6) Having demographic data of the patient populations attended to during the study period, of the quantitative evaluation of the Nursing workload and of the incidence of adverse events may have helped establish more precise analysis of the data and of the impact of AASTRE (Additional file 1).

In conclusion, our results suggest that the AASTRE were associated with improved structure, process and outcome indicators. In addition, this tool allows simultaneously translating medical evidence to clinical practice, reducing errors of omission, and also allows assessing quality through process indicators.

ICU:

intensive care unit

AASTRE:

Análisis Aleatorios de Seguridad en Tiempo Real

CRRT:

continuous renal replacement techniques

SCS:

Safety Climate Survey

SAQ-ICU:

Safety Attitude Questionnaire-ICU model

PNR:

patient:nurse ratio

PPR:

patient:physician ratio

SOFA:

Sequential Organ Failure Assessment

NS:

no significant differences

IPR-AASTRE:

improvement proportion related to the AASTRE

IPR-AASTRE-G:

improvement proportion related to the AASTRE globally

IPR-AASTRE-B:

improvement proportion related to the AASTRE for each block of variables

MV:

mechanical ventilation

CRB:

central venous catheter-related bacteraemia

VAP:

ventilator-associated pneumonia (VAP)

CVC:

central venous catheter

All authors contributed to study conception and design, data analysis, and drafting the manuscript. MOC contributed to data analysis and statistical analysis. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

The authors added the database of the present study as supplementary material.

Ethics approval and consent to participate

The study was approved by the Ethics and Clinical Research Committee of each investigating centre. It was deemed unnecessary to obtain informed consent.

Funding

This study was supported by Grants from the Fondo de Investigación Sanitaria (Institute of Health Carlos III from Spain, FIS Grants, Project PI11/02311) and from Fundación Ricardo Barri Casanovas. FEDER.2014 SGR 926.

Affiliations

1. Intensive Care Unit, Hospital Universitario Joan XXIII, Tarragona, Spain

   • María Bodí
   • , Iban Oliva
   • , Maria Carmen Gilavert
   •  & Gonzalo Sirgo

2. Instituto de Investigación Sanitaria Pere Virgili, Rovira i Virgili University, Tarragona, Spain

   • María Bodí
   • , Montserrat Olona
   •  & Gonzalo Sirgo

3. Centro de Investigación Biomédica en Red de Enfermedades Respiratorias (CIBERES), Instituto de Salud Carlos III, Madrid, Spain

   • María Bodí

4. Intensive Care Unit, Hospital Universitario de Torrejón, Torrejón de Ardoz, Madrid, Spain

   • Maria Cruz Martín
   •  & Carlos Muñoz

5. Department of Preventive Medicine, Hospital Universitario Joan XXIII, Tarragona, Spain

   • Montserrat Olona

Corresponding author

Correspondence to Iban Oliva.

Reprints and Permissions