External validity of Adult Sepsis Event’s simplified eSOFA criteria: a retrospective analysis of patients with confirmed infection in China

Abstract

Background

The US Centers for Disease Control and Prevention (CDC) recently released simplified eSOFA organ dysfunction criteria of Adult Sepsis Event for sepsis surveillance in the US. Our study aimed to compare the prevalence, characteristics, and outcomes of sepsis patients identified by eSOFA criteria versus Sequential Organ Failure Assessment (SOFA) Score (Sepsis-3) and assess the external validity of eSOFA criteria in China.

Methods

We conducted a retrospective cohort study of adult residents of Yuetan Subdistrict, Beijing, China, who were hospitalized from July 1, 2012 to June 30, 2014. Among patients with infection, sepsis was identified if there was a concurrent rise in SOFA score by 2 or more points (Sepsis-3) or the presence of 1 or more eSOFA criteria: vasopressor initiation, mechanical ventilation initiation, doubling in creatinine, doubling in bilirubin to 2.0 mg/dL or above, 50% or greater decrease in platelet count to less than 100 cells/μL, or lactate equal to or above 2.0 mmol/L. Areas under the receiver operating characteristic curves (AUROCs) for in-hospital mortality were compared between sepsis patients detected by the two criteria, adjusting for baseline characteristics.

Results

Of 1716 hospitalized patients with infection, 935 (54.5%) met Sepsis-3 criteria, 573 (33.4%) met eSOFA criteria, while 475 (27.7%) met both criteria. Demographic and clinical characteristics of sepsis patients meeting Sepsis-3 or eSOFA criteria were similar. In-hospital mortality was higher with eSOFA criteria versus Sepsis-3 (46.6% vs. 32.0%, p < 0.001). eSOFA criteria had high PPV (82.9%), but low sensitivity (50.8%) for the diagnosis of Sepsis-3. Patients meeting both criteria had the highest in-hospital mortality rate (52.8%, all p < 0.001), while patients who only met eSOFA criteria had higher mortality rate than those meeting Sepsis-3 alone (16.3% vs. 10.4%, p = 0.097). The predicted probability for in-hospital mortality was higher with eSOFA criteria versus Sepsis-3 (AUROC 0.830 vs. 0.795, p = 0.001) adjusting for baseline characteristics.

Conclusions

The CDC Adult Sepsis Event’s eSOFA criteria identify a smaller, more severely ill cohort of sepsis patients with similar demographic and clinical characteristics as the more complex Sepsis-3 SOFA score. These results suggest similar performance of eSOFA criteria across diverse populations, with low sensitivity and high specificity for the diagnosis of Sepsis-3.

Introduction

Sepsis remains the leading cause of death in critically ill patients, with over 1.7 million adult sepsis cases annually in the US which contribute to 270,000 deaths [1]. In China, standardized sepsis-related mortality rate was 66.7 deaths per 100,000 population, producing a national estimate of 1,025,997 sepsis-related deaths in 2015 [2].

The Third International Consensus Definitions Task Force defined sepsis as a “life-threatening organ dysfunction due to a dysregulated host response to infection”, and recommended use of an acute increase in Sequential Organ Failure Assessment (SOFA) Score by 2 or more points as the working definition (Sepsis-3) to identify the presence of organ dysfunction [3]. However, alternative sepsis criteria serve different purposes, including clinical care, research, surveillance, and quality improvement and audit [4]. Sepsis-3 based on SOFA score was chosen for clinical care due to their superior content and criterion validity as well as good timeliness [5], but did not perfectly suit the purpose of surveillance since many components are not routinely or consistently recorded [6]. Moreover, it is particularly difficult to calculate SOFA score from the electronic health records (EHRs) which is not dedicated for this purpose.

In 2018, the US Centers for Disease Control and Prevention (CDC) proposed the criteria of Adult Sepsis Event, based on organ dysfunction criteria (eSOFA) analogous to SOFA score (Sepsis-3) [7]. It was developed for retrospective surveillance using objective data that can be directly obtained from EHRs. Rhee et al. have compared sepsis patients detected by eSOFA criteria with those identified by Sepsis-3 and validated the use of eSOFA for sepsis surveillance in the US [1, 6]. It is important to assess the external validity of eSOFA criteria in other countries. However, the predictive validity only represented one, although the most commonly studied, of the six domains of usefulness for sepsis criteria [4, 8]. As a practical, simplified adaptation of Sepsis-3 that was newly proposed for consistent, automated sepsis surveillance, it is also crucial to understand the prevalence, clinical characteristics and outcomes of sepsis patients who were missed by eSOFA criteria (i.e., false negatives) and those misdiagnosed as sepsis (i.e., false positives).

In this retrospective study, we compared the prevalence, characteristics, and outcomes of sepsis patients detected using eSOFA (CDC Adult Sepsis Event) versus SOFA score (Sepsis-3) in a database of patients with Sepsis-1 from a subdistrict of Beijing. In addition, we also investigated characteristics and outcomes of false negatives and false positives based on eSOFA criteria. We hypothesized that CDC Adult Sepsis Event’s simplified eSOFA criteria could perform comparably as Sepsis-3 in terms of detecting sepsis patients and predicting mortality, which supports its use as a practical tool for sepsis surveillance.

Methods

Study design, data source, and definitions

This was a retrospective analysis of a database of 1716 patients fulfilling Sepsis-1 criteria, with data source and definitions of the study described in a previous study [9]. We conducted a retrospective cohort study of all adult residents (≥ 18 years old) of Yuetan Subdistrict, Beijing, China, who were hospitalized from July 1, 2012 to June 30, 2014. Medical records of these patients were identified from the hospital discharge database of Beijing Public Health Information System and manually reviewed independently by any two of three investigators each with more than 5 years of ICU working experience. Any disagreement was resolved by discussion. Final decision was made by the steering committee (XM, YA, and BD) if consensus could not be reached.

Patients with any of the following definitions were identified as infected. Community-acquired infection was identified based on clinical, imaging, and microbiologic parameters, whereas nosocomial infection was diagnosed according to the set of standardized definitions of the CDC [10]. Microbiologically documented infection was confirmed by positive cultures of blood or body fluid from a site of suspected infection, and patients with the presence of gross purulence or an abscess (anatomical and/or by imaging and/or histologic evidence), but without a microbiologic documentation, were considered to have clinically documented infection.

The US CDC released the Adult Sepsis Event (eSOFA) as simpler criteria that include the same organ systems as SOFA score except replacing Glasgow Coma Score (GCS) by lactate greater than or equal to 2.0 mmol/L [7]. The eSOFA criteria include the following organ dysfunctions: (1) vasopressor initiation; (2) initiation of mechanical ventilation; (3) doubling of serum creatinine or decrease by 50% of estimated glomerular filtration rate (eGFR) relative to baseline, excluding patients with end-stage renal disease; (4) total bilirubin ≥ 2.0 mg/dL and increase by 100% from baseline; (5) platelet count < 100 cells/μL and ≥ 50% decline from baseline (baseline must be ≥ 100 cells/μL); (6) serum lactate ≥ 2.0 mmol/L. For patients with infection, we calculated maximum eSOFA and SOFA score based on retrieved clinical data until 72 h after hospital admission (for those who were admitted due to infection) or onset of infection (for those who developed infection during hospitalization). We identified a hospital admission as having sepsis if there was infection and concurrent organ dysfunction defined by either the presence of 1 or more eSOFA criteria [1, 6, 7] or a rise in SOFA score by 2 or more points (Sepsis-3) [3].

Missing data imputation for eSOFA or SOFA score were performed based on relevant information in the medical records, surrogate markers, or data obtained before and after data collection date. If none of these were available, we recorded the missing variable as zero for the corresponding category of organ dysfunction in the final analysis. For example, we considered free text such as no jaundice in the medical records as surrogates for normal serum bilirubin level, or consciousness as indication of normal mentation. Moreover, in cases without arterial blood gas, we substituted SpO₂/FiO₂ ratio for PaO₂/FiO₂ ratio [11]. However, we did not perform missing data imputation for lactate due to the lack of reliable surrogate markers.

Statistical analysis

We examined the prevalence, characteristics, and outcomes of sepsis patients defined by either eSOFA or Sepsis-3 criteria. Crude mortality rates were compared using two-sample z-test. The sensitivity, specificity, and positive predictive value (PPV) of eSOFA criteria were calculated comparing to Sepsis-3 criteria. The agreement between sepsis patients identified by SOFA and eSOFA criteria was examined using Cronbach’s alpha [3, 12].

The predictive values of Sepsis-3 and eSOFA criteria for in-hospital mortality were compared by the area under the receiver operation characteristics (AUROC) curves with DeLong method, with and without adjustment for covariates in multivariate logistic regression analysis. In addition, in-hospital mortality rates of patient groups classified by Sepsis-3 and eSOFA criteria (i.e., Sepsis-3−/eSOFA−, Sepsis-3+/eSOFA−, Sepsis-3−/eSOFA+ and Sepsis-3+/eSOFA+) were also compared by multivariate logistic regression analysis. Potential risk factors added into the model included demographics (age and gender), body mass index (BMI), Charlson Comorbidity Index, and characteristics of infection. Age was categorized into three categories (18–64, 65–84, and ≥ 85 years), because the assumption of linearity would be violated if age was included in the model as a continuous variable [13, 14]. Akaike information criterion (AIC) was used to measure the relative quality of the models.

Continuous variables were presented as median and interquartile range (IQR). Categorical variables were presented as a percentage of the group from which they were derived, and compared by the use of Chi-square test or Fisher’s exact test. All comparisons were unpaired and all tests of significance were two-tailed. Analyses were conducted using SPSS version 22. A p value < 0.05 was considered as statistically significant.

Ethical approval

This study was approved by the ethics committee of Peking Union Medical College Hospital and informed consent was waived. This study was registered at ClinicalTrials.gov, with registration number NCT02285257.

Results

Prevalence, characteristics, and in-hospital mortality of sepsis patients defined by eSOFA or Sepsis-3 criteria

During the study period, 22,552 Yuetan residents were admitted into any of the 111 hospitals within the Beijing Public Health Information System, of whom the medical records of 21,191 admissions were manually reviewed. We were unable to review the medical records of the other 1361 admissions either because of missing records (n = 277) or refusal by the hospitals (n = 1084). A total of 1716 patients meeting Sepsis-1 criteria were identified from 3449 patients with infection, and were included in the final analysis.

Among the 1716 infected patients, 935 (54.5%) met Sepsis-3 criteria, 573 (33.4%) met CDC Adult Sepsis Event eSOFA criteria, while 475 (27.7%) met both criteria. The agreement between eSOFA and Sepsis-3 criteria was moderate with Cronbach’s alpha 0.56. Frequency of missing variables and missing data imputation for SOFA and eSOFA criteria is shown in Additional file 1: Table S1.

Demographics, comorbidities, and clinical characteristics of sepsis patients meeting Sepsis-3 or eSOFA criteria were generally similar (Table 1). Compared with those who did not meet Sepsis-3 criteria, patients defined by Sepsis-3 criteria were older, more likely to be male, and prone to be complicated with chronic heart, pulmonary, or renal diseases. Similar differences were also found comparing patients meeting eSOFA criteria or not (Table 1). Pneumonia and intra-abdominal infections were the most common sites of infection for both sets of septic patients. Respiratory and coagulation dysfunction were the most common organ dysfunctions in sepsis patients defined by Sepsis-3, whereas respiratory dysfunction and elevated lactate were the most common organ dysfunctions in patients meeting eSOFA criteria (Additional file 1: Table S2).

Table 1 Characteristics of infected patients with or without sepsis defined by Sepsis-3 or eSOFA criteria

Crude in-hospital mortality rates were higher in patients meeting eSOFA criteria than those meeting Sepsis-3 criteria (46.6% vs. 32.0%; p < 0.001). The predictive value of eSOFA criteria for in-hospital mortality was significantly higher than that of Sepsis-3 criteria, with (AUROC 0.830 [95% CI 0.812–0.848] vs. 0.795 [0.775–0.814]; p = 0.001) and without (0.762 [0.742–0.782] vs. 0.690 [0.668–0.712]; p < 0.001) adjustment for demographics, BMI, Charlson Comorbidity Index and site of infection.

Prevalence, characteristics, and in-hospital mortality of Sepsis-3+/eSOFA−, Sepsis-3−/eSOFA+, and Sepsis-3+/eSOFA+ patients

Infected patients who met Sepsis-3 but not eSOFA criteria (n = 460) were less likely to have comorbidities such as cancer, chronic respiratory dysfunction, and chronic heart disease compared with patients who met both criteria (Table 2). In terms of organ dysfunctions flagged by SOFA score, Sepsis-3+/eSOFA− patients tended to have mild hypoxia that did not require mechanical ventilation, mild coagulation dysfunction, or elevated creatinine that did not reach twice of baseline creatinine (Additional file 1: Table S2).

Table 2 Characteristics of sepsis patients categorized by overlap of Sepsis-3 and eSOFA criteria (Adult Sepsis Event)

Meanwhile, infected patients who met eSOFA but not Sepsis-3 criteria (n = 98) were more likely to be complicated by cancer or chronic respiratory dysfunction compared with sepsis patients meeting both criteria (Table 2). Among the 98 eSOFA+/Sepsis-3− patients, 30 (30.6%) had elevated lactate alone without other organ dysfunctions (Additional file 1: Table S2). Mechanical ventilation was initiated for 24 (24.5%) patients, whose PaO₂/FiO₂ were not measured (n = 9) or did not fall under 300 (i.e., respiratory SOFA score ≤ 1) (n = 15). Detailed description of eSOFA/SOFA organ dysfunctions of eSOFA+/Sepsis-3− patients is presented in Additional file 1: eResults.

Sepsis patients who met both Sepsis-3 and eSOFA criteria tended to have elevated lactate and high rates of cardiovascular and respiratory dysfunction (Additional file 1: Figure S1). Patients in Sepsis-3−/eSOFA− group had the lowest mortality rate (5.6%, all p < 0.001). In addition, patients in Sepsis-3−/eSOFA+ group had a higher mortality rate than those in Sepsis-3+/eSOFA− group (16.3% vs. 10.4%; p = 0.097), with adjusted odds ratio (OR) of 2.45 (95% CI 1.27–4.73) and 2.09 (95% CI 1.35–3.25), respectively. In comparison, patients in Sepsis-3+/eSOFA+ group had the highest mortality rate (52.8%, all p < 0.001), with adjusted OR of 17.20 (95% CI 11.52–25.67) (Additional file 1: Table S3).

The sensitivity of Adult Sepsis Event eSOFA Criteria for Sepsis-3 diagnosis was 50.8%, the specificity was 87.5%, and PPV was 82.9%.

Discussion

In this population-based cohort in China, we found that CDC Adult Sepsis Event eSOFA criteria identified a smaller group of sepsis patients with higher mortality compared to Sepsis-3 criteria. These findings were consistent with previous study in US hospitals that reported lower prevalence (4.4% vs. 6.1%) and higher mortality (17.1% vs. 14.4%) of sepsis patients defined by eSOFA versus Sepsis-3 criteria [6].

In our study, eSOFA criteria had high PPV (82.9%). Only 98 patients (false positive rate 12.5%) were misdiagnosed as sepsis by eSOFA criteria, with hyperlactatemia without evidence of concurrent organ dysfunctions being the major cause. Moreover, the mortality rate of these patients was significantly higher than those fulfilling neither Sepsis-3 nor eSOFA criteria (16.3% vs. 5.6%) [15,16,17]. This indicates that SOFA score alone is unable to detect all infected patients with high risk of mortality [18], and further studies are needed to assess whether additional screening parameters such as lactate could be helpful. In comparison, eSOFA criteria identified more severely ill patients, possibly by including invasive procedures (such as vasoactive agents or invasive mechanical ventilation) in the criteria, although they might not be the same patients diagnosed as septic at the bedside. Meanwhile, our study presented a severely ill cohort of infected patients with older age, more pneumonia (60.3%), higher rates of mechanical ventilation (13.6%) and vasopressor use (7.9%), which might explain the high PPV.

The sensitivity of eSOFA criteria (50.8%) was considerably reduced by patients with mild hypoxia that did not reach the eSOFA point of mechanical ventilation, those with mild thrombocytopenia that did not reach a decrease of platelets by 50% or more from baseline, as well as those with elevated creatinine that was lower than the doubling of baseline. As a result, eSOFA criteria might miss the diagnosis of septic patients who are less severely ill but with significant risk of in-hospital mortality (adjusted OR 2.09, 95%CI 1.35–3.25), questioning the potential use of eSOFA alone as a surveillance tool for sepsis [6].

Our study has some strengths. First, no study has evaluated the external validity of eSOFA criteria or investigated the characteristics and outcomes of infected patients missed and misdiagnosed by eSOFA criteria. Moreover, we manually reviewed the medical records of all patients to obtain relevant data and made diagnosis of infection and sepsis using standardized definitions, whereas previous studies used EHR-based proxies as culture orders and antibiotic administrations for infection [3, 6].

Our study has several limitations. First, this study is a secondary analysis of a database that was not originally designed for the study purpose. Among the 3449 patients with infection in the original cohort, we only collected data from 1716 patients who met Sepsis-1 criteria. This might introduce selection bias to the current study. However, Luo et al. reported that, among 38 infected patients who did not fulfill Sepsis-1 criteria, only 5 were diagnosed as Sepsis-3 [18]. Therefore, it was unlikely that addition of these patients in our analysis would change the major results. Second, the mortality rates of septic patients were significantly higher than those in Rhee et al.’s study. Older age (median age 82 for eSOFA+ patients, and 81 for Sepsis-3+ patients in our study) and higher rate of pneumonia might cause the difference since they were proved as independent risk factors for mortality of sepsis patients [4, 5, 19, 20]. Meanwhile, mortality rates of septic patients vary in previous studies by factors including age, sex, comorbidities, and acuity of illness [2, 19,20,21,22]. Third, our cohort represented a patient population significantly different from the original patient population from which eSOFA criteria were developed, as suggested by older age, more pneumonia, higher rate of mechanical ventilation, and higher mortality. Although our results might not be generalized to other patient populations, it clearly demonstrated the same advantages and disadvantages of eSOFA criteria, i.e., external validity, in different settings and patients [6]. Fourth, some data necessary for the calculation of SOFA and/or eSOFA score were missing, particularly with high proportion of missing data from lactate values. However, the frequency of missing data in our cohort was comparable to that in previous studies [3, 6]. As a matter of fact, this might be regarded as limitations of the scoring systems, rather than limitations of the current and previous studies, because both eSOFA and SOFA scoring systems contained variables such as lactate and, possibly, bilirubin that were not routinely monitored in general wards. Last, due to the lack of gold standard for diagnosis of sepsis [4, 5, 23, 24], concordance between eSOFA and Sepsis-3 criteria, rather than sensitivity and specificity, should be reported. However, in order to highlight the limitations of applying eSOFA criteria as a surrogate of SOFA score, as well as to compare our results with those in previous study [6], we still calculated sensitivity, specificity, and PPV of eSOFA criteria compared to Sepsis-3.

Conclusion

In conclusion, we found that, similar to the study in the US hospitals, the CDC Adult Sepsis Event’s simplified eSOFA organ dysfunction criteria identify a smaller cohort of sepsis patients with similar demographic and clinical characteristics as those identified using Sepsis-3 SOFA score, but with higher risk of death. These results suggest similar performance of eSOFA criteria across diverse populations. However, the poor prognosis of patients with Sepsis-3 who are missed by eSOFA criteria might limit the use of eSOFA criteria as a surveillance tool for sepsis.