- Open Access
New perspectives to improve critical care benchmarking
Annals of Intensive Care volume 8, Article number: 17 (2018)
In the last decades, experts and medical societies have strongly recommended the measurement of quality indicators and the assessment of intensive care unit (ICU) performance [1, 2]. This recommendation is based on the concept that measurement will generate transparency of results for multiple stakeholders and be a robust source of targets for quality improvement to be translated into improved clinical outcomes. Apart from the measurement of the results of a single ICU over time, reproducible and risk-adjusted outcomes are used to generate performance comparisons  in benchmarking projects for ICUs as employed in hospitals networks and national registries .
What’s next for critical care benchmarking?
The widespread use of information technology (IT) in healthcare altogether with more sophisticated modeling statistics and newer techniques such as machine learning have potential implications for the next steps in benchmarking (Fig. 1). From the IT perspective, increased interoperability of medical devices, electronic health records (EHRs) and information systems will change the amount and speed of data acquisition and presentation to healthcare professionals, additionally it will expand access to data on processes of care and patient-centered outcomes. Moreover, the widespread use of mobile and wearable devices will allow access to information regarding quality of life, activities of daily living and functional status reported by the patients. In a recent study involving 60,000 ICU admissions, functional capacity prior to ICU admission was not only independently associated with outcomes but also improved the prognostic ability of SAPS3 . Web-based centralized systems and mobile apps will play a major role in this scenario as shown in recent studies of post-hospital follow-ups  and survivors of critical illness. This should allow a better evaluation of the effectiveness of ICU care on long-term survival, readmissions, social and labor insertion and cognitive function. Also, as the healthcare systems evolve with the increasing role of long-term acute care facilities, hospices and institutions dedicated to rehabilitation, outcomes such as hospital length of stay may become obsolete for benchmarking. Thus, thinking of a full cycle of an episode of care, outcomes such as “home-to-home” time can be pursued and benchmarked.
Finally, as machine learning and cluster analysis are applied in critical care, they will also facilitate benchmarking. These techniques will enable clinicians to better understand the clinical profiles of critically ill populations without being necessarily categorized a priori by a diagnosis or a level of severity. In studies performed in large databases of real-world patients  or clinical trials , patients with the same diagnosis and comparable severity of illness have different profiles, response to treatments and outcomes. Therefore, an expected next step for these applications will be to test them for benchmarking purposes as they could reduce the heterogeneity of the critically ill populations enabling even more objective comparisons of outcomes.
Actionable data: a way to decrease practice variation
The expression “Actionable data” has been used in the context of quality improvement initiatives. Let’s use the low-tidal volume (Vt) in ARDS, a practice associated with reduced mortality and shorter ICU length of stay (LOS), as an example. At patient level, daily checklists using mobile devices could show the current recommendation (Vt ≤ 6 mL/Kg IBW) and capture the patients’ current tidal volume, as well as the average data in the ICU and in the country/region thus helping to improve adherence to good standards of care. The aggregate of all the information captured by the checklist could be used at unit level for a better understanding of its overall adherence rate and comparing it with similar ICUs and ICU populations. Data available in a dashboard allows ICUs to compare the overall rates and specific outcomes (ICU LOS, duration of MV, VAP rates) and also to analyze what are the patterns of ICUs that are top ranked. These positive outliers could be considered as “positive deviants.” This methodology is used with success in other medical areas and briefly consists of understanding what are the characteristics of patients (e.g., case-mix, severity, etiology of ARF), structure (e.g., staffing patterns, types of ICU) and processes of care (e.g., sedation practices, weaning protocols, ventilator settings) of those that achieve better results. This may provide a road map for a PDCA cycle based on benchmarking of practices that are associated with better performance.
Through national ICU registries, benchmarking would support the implementation of public health policies in critical care and reduce practice variation and to increase efficiency. Currently, several countries have successfully implemented national ICU registries. Some initiatives as those in the UK (www.icnarc.org), Netherlands (www.stichting-nice.nl), Brazil (www.utisbrasileiras.com) and Australia–New Zealand (www.anzics.com.au) provide open access to epidemiologic data on critical illness, outcomes and resource utilization in these countries.
With the increasing implementation of EHRs and the availability of multiple data sources, projects are starting to collect information on processes of care. The availability of adherence to best practices, outcomes and evidence-based recommendations may decrease practice variation and improve outcomes. The NICE project has recently introduced this concept in relevant clinical areas such as pain control, red blood cell transfusion triggers or the use of antimicrobials where it is feasible to implement and perhaps reduce the evidence to practice gaps in implementation .
Also, increasingly, our community questions the impact of traditional clinical trials on our ability to change practice and improve patient outcome. The IT components of modern healthcare organizations and benchmarking projects may foster the development of “platform” trials, allowing to focus on diseases or syndromes, comparing multiple interventions within different domains of treatment . The next leap could be the advent of a new kind of randomized, embedded, multifactorial, adaptive platform (REMAP) trials that will leverage the strengths of big data while retaining the advantages of randomization . Perhaps combining elements of clinical research and continuous quality improvement programs, these trials may fuse clinical research and quality improvement.
In conclusion, benchmarking is a reliable tool for quality improvement. The recent advances in IT and data science will change the way ICU benchmarking is provided, making information widely available and actionable to improve patient’s outcomes.
intensive care unit
electronic health records
length of stay
acute respiratory failure
Netherlands Intensive Care Evaluation
randomized, embedded, multifactorial, adaptive platform trial
Rhodes A, Moreno RP, Azoulay E, Capuzzo M, Chiche JD, Eddleston J, Endacott R, Ferdinande P, Flaatten H, Guidet B, Kuhlen R, León-Gil C, Martin Delgado MC, Metnitz PG, Soares M, Sprung CL, Timsit JF, Valentin A. Task force on safety and quality of European Society of Intensive Care Medicine (ESICM). Prospectively defined indicators to improve the safety and quality of care for critically ill patients: a report from the task force on safety and quality of the European Society of Intensive Care Medicine (ESICM). Intensive Care Med. 2012;38(4):598–605.
Flaatten H. The present use of quality indicators in the intensive care unit. Acta Anaesthesiol Scand. 2012;56(9):1078–83.
Salluh JI, Soares M, Keegan MT. Understanding intensive care unit benchmarking. Intensive Care Med. 2017. https://doi.org/10.1007/s00134-017-4760-x.
Cooke CR, Iwashyna TJ. Using existing data to address important clinical questions in critical care. Crit Care Med. 2013;41(3):886–96.
Zampieri FG, Bozza FA, Moralez GM, Mazza DD, Scotti AV, Santino MS, Ribeiro RA, Rodrigues Filho EM, Cabral MM, Maia MO, D’Alessandro PS, Oliveira SV, Menezes MA, Caser EB, Lannes RS, Alencar Neto MS, Machado MM, Sousa MF, Salluh JI, Soares M. The effects of performance status one week before hospital admission on the outcomes of critically ill patients. Intensive Care Med. 2017;43(1):39–47.
Shaw BE, Brazauskas R, Millard HR, Fonstad R, Flynn KE, Abernethy A, Vogel J, Petroske C, Mattila D, Drexler R, Lee SJ, Horowitz MM, Rizzo JD. Centralized patient-reported outcome data collection in transplantation is feasible and clinically meaningful. Cancer. 2017. https://doi.org/10.1002/cncr.30936.
Vranas KC, Jopling JK, Sweeney TE, Ramsey MC, Milstein AS, Slatore CG, Escobar GJ, Liu VX. Identifying distinct subgroups of ICU patients: a machine learning approach. Crit Care Med. 2017;45(10):1607–15.
Famous KR, Delucchi K, Ware LB, Kangelaris KN, Liu KD, Thompson BT, Calfee CS. ARDS network. Acute respiratory distress syndrome subphenotypes respond differently to randomized fluid management strategy. Am J Respir Crit Care Med. 2017;195(3):331–8.
De Lange DW, Dongelmans DA, De Keizer NF. Small steps beyond benchmarking. Rev Bras Ter Intensiva. 2017;29(2):128–30.
Berry SM, Connor JT, Lewis RJ. The platform trial: an efficient strategy for evaluating multiple treatments. JAMA. 2015;313(16):1619–20.
Angus DC. Fusing randomized trials with big data: The key to self-learning health care systems? JAMA. 2015;314(8):767–8.
All authors read and approved the final manuscript.
Drs. Salluh, Soares and Reis are founders and equity holders at Epimed Solutions, the provider of a cloud-based healthcare analytics and performance evaluation software. Prof Chiche received honorarium from GE Healthcare for lectures at various medical congresses.
Availability of supporting data
Consent for publication
All authors approved and consent the present publication.
Ethics approval and consent to participate
The present manuscript had no source of funding. Drs. Salluh and Soares are supported in part by individual research grants from CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPERJ (Fundação Carlos Chagas Filho de Amparo à Pesquisa do Rio de Janeiro).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Salluh, J.I.F., Chiche, J.D., Reis, C.E. et al. New perspectives to improve critical care benchmarking. Ann. Intensive Care 8, 17 (2018). https://doi.org/10.1186/s13613-018-0363-0
- Big data
- Intensive care
- Quality improvement
- Scoring systems