Estimation of the effect of vaccination in critically ill COVID-19 patients, analysis using propensity score matching

Abstract

Background

Vaccination helped in reducing mortality and disease severity due to COVID-19. Some patients can develop breakthrough infections. The effect of vaccination in critically ill patients admitted with breakthrough infections is not well studied. We designed a study to estimate the effect of vaccination on ICU mortality in critically ill COVID-19 patients by using propensity score matching.

Methods

We included patients from 15th June 2020 to 31st December 2021. Inclusion criteria were unvaccinated and vaccinated COVID-19 patients requiring intensive care unit (ICU) admission. The institutional ethics committee approval was obtained (institutional ethics committee, IEC 08/2023, Clinical trial registry, India CTRI/2023/01/049142). The primary outcome was ICU mortality. The secondary outcomes were the length of ICU stay and duration of mechanical ventilation. We used multivariable logistic regression (MLR) and propensity score matching (PSM) for the statistical analysis.

Results

Total of 667 patients (79.31%) were unvaccinated and 174 (20.68%) vaccinated. The mean age was 57.11 [standard deviation (SD) 15.13], and 70.27% were males. The ICU mortality was 56.60% [95% confidence interval (CI) 53.24–60%]. The results of MLR and PSM method showed that vaccinated patients were less likely to be associated with mortality [adjusted odds ratio (AOR), 95% CI using logistic regression: 0.52 (0.29, 0.94), and by propensity score matching: 0.83 (0.77, 0.91)].

Conclusion

The findings of this study support COVID-19 vaccination as an effective method for reducing case fatality not only in the general population but also in critically ill patients, and it has important public health implications.

Introduction

The COVID-19 pandemic has caused varied presentations, primarily affecting the respiratory system leading to pneumonia. The disease had different severity ranging from mild or asymptomatic infection to a severe disease requiring intensive care unit admission [1]. The intensivists used different therapies during the first wave. The use of steroids helped in reducing mortality in patients requiring oxygen therapy [2]. Vaccination was one of the effective measures to contain the pandemic [3, 4]. Breakthrough infections can occur despite vaccination. We defined a breakthrough infection as a COVID-19 infection resulting after the first or second dose of vaccination. The presence of comorbidities and severe COVID-19 infection resulted in the requirement for ICU care. There is a paucity of data on the effect of vaccination in the ICU population.

The population-based study from Israel showed two doses of the BNT162b2 vaccine reduced symptomatic as well as asymptomatic COVID-19 infections, and breakthrough infection was less severe with reduced hospitalisation and lower mortality [5]. A similar study from the Calabria region of Italy on BNT162b2 vaccination showed significantly lesser mortality (24.3%) in the vaccinated as compared to unvaccinated patients (38.5%) [6]. A recent study from Turkey showed BioNTech and Sinovac vaccination reduced the severity of illness, need for invasive ventilation, and mortality. The hospital mortality was 41.5% in vaccinated as against 64.9% in the unvaccinated patients [7]. The UK app-based study from London showed frailty was one of the risk factors associated with breakthrough infections in the above 60 years age group after the first dose. The study participants had received BNT162b2, ChAdOx1nCoV19, or mRNA-1273 vaccine. The symptoms were less frequent in the vaccinated individuals [8].

Grasselli et al., showed vaccination with mRNA or adenoviral vector vaccine reduced ICU admissions for COVID-19 infection. The vaccinated patients were older and had more comorbidities. There was no association between vaccination status and mortality [9]. Similar observations were seen in the studies from Switzerland and Spain [10, 11]. The Australian study showed higher ICU and hospital mortality in vaccinated patients. After adjusting for the covariates, the mortality in the vaccinated and unvaccinated patients was similar [12]. As the results from the available literature are inconsistent, there is a need to know whether vaccination reduced the mortality in critically ill patients who developed breakthrough infections after vaccination. We aimed to study the effectiveness of vaccination in critically ill COVID-19 patients who developed breakthrough infection by propensity score matching.

Methods

The institutional ethical committee approval was obtained, IEC/08/2023, CTRI/2023/01/049142, and data were retrieved. Patients were included from 15th June 2020 to 31st December 2021. The current study includes data from 2 multicentre published studies [13, 14], consisting of 841 patients. The epidemiology and ventilation characteristics of confirmed cases of severe COVID-19 pneumonia admitted in intensive care unit (EPIC19) study described the epidemiology and ventilation characteristics of the unvaccinated COVID-19 patients [13]. The second study, the Postcovac-covid group, described the characteristics of the patients who developed breakthrough infections after COVID-19 vaccination. The patients of the Postcovac-covid study had received either ChAdOx1 nCov19 (Covishield) or BBV 152 COVID-19 (Covaxin) vaccine. The median time from vaccination to the hospitalisation was 33.5 days [14]. The baseline characteristics like age, gender, comorbidities, arterial blood gas (ABG) pH, and Pao2/Fio2 ratio (PF ratio) were collected. The ABG pH corresponds to the worst pH value during the first 24 h of ICU admission. The PF ratio defined as the ratio of partial pressure of oxygen (PaO2) divided by fraction of inspired oxygen (Fio2).

The acute physiology, age, and chronic health evaluation (APACHE II) score and sequential organ failure score (SOFA) were collected. The APACHE II considers various laboratory and clinical parameters and the presence of acute and chronic diseases [15]. It provides information about the severity of the illness and estimates the mortality. The SOFA score evaluates organ failure involving six organ systems and estimates the mortality [15, 16]. The primary outcome was the effect of vaccination on ICU mortality. The secondary outcomes were the length of ICU stay and duration of mechanical ventilation.

Statistical analysis

Continuous variables represented as mean (standard deviation SD) for the normally distributed variables or median with 25th and 75th percentiles for non-normal variables. The categorical variables presented as numbers with percentages. Initial analyses were performed using independent t-test and Mann–Whitney U test as applicable, to compare the clinical characteristics such as age, APACHE II score, SOFA score, ABG pH, and PF ratio, duration of mechanical ventilation, and length of ICU stay between vaccinated and unvaccinated patients. The association between categorical variables and vaccination status was assessed using the chi-squared test. The significant variables in the univariate analysis and clinically relevant were considered for multivariable logistic regression analysis (MLR). The MLR was performed to compare the ICU mortality between vaccinated and unvaccinated patients adjusted for covariates. Propensity score matching (PSM) was performed.

Propensity score matching (PSM)

PSM is to estimate the effect of treatment when randomisation is not possible. In randomised controlled trials, random assignment of patients into intervention and control groups balances individuals for all the observed and unobserved characteristics. Whereas in observational studies, the treatment assignment is not random and causes an imbalance in the baseline characteristics, leading to a selection bias. PSM is the recommended statistical method for balancing the measured covariates between treated and control groups.

The propensity score is a balancing score. In PSM, treatment and control patients are paired based on similar propensity scores and possibly other covariates. It is the probability of treatment assignment conditional on the observed baseline covariates, e_k = Pr(Z_k = 1|X_k). In this equation, for subject k, Z_k = 1 is the treatment assigned, and X_k is the vector of observed covariates. We had a higher number of unvaccinated than vaccinated patients; hence, we used the matching procedure of PSM as the recommended method [17,18,19]. The vaccinated and unvaccinated patients were matched on the estimated propensity score. For PSM, covariates that are potentially related to the outcome were included. The covariates selected were age, gender, comorbidities, APACHE II score, need for invasive ventilation, ABG pH, need for renal replacement therapy, and use of steroids.

After performing the PSM using matching method, a check for the balance of individual covariates across vaccinated and unvaccinated patients for the estimated propensity score was performed using a Kernel density plot. Each vaccinated individual was assigned a weight of one. In order to match the individuals between vaccinated and unvaccinated groups, weighted composite of comparison observations was considered. The comparison patient’s propensity scores were weighted within a range of propensity scores according to the distance, and they were from the vaccinated subjects [18]. The observations outside of the common support range and showed no overlap between vaccinated and unvaccinated patients were excluded. Out of 841 patients, 718 patients were in the range of common support, hence included in the further analysis. The Kernel density plot before matching and after matching of the covariates was plotted for vaccinated and unvaccinated patients. The effect of vaccination was estimated and reported using the average treatment effect (ATE) for the entire sample. The ATE is estimated from a sample using a comparison in mean outcomes for treated and untreated units. The statistical analysis was performed using STATA™ (Version 14, College Station TX) software. The p value < 0.05 was considered as statistically significant.

Results

A comparison of baseline characteristics between vaccinated and unvaccinated groups is presented in Table 1. We included 841 patients in the analysis, 667 (79.31%) unvaccinated and 174 (20.68%) vaccinated patients (Table 1).

Table 1 Baseline characteristics in the vaccinated and non-vaccinated population

The mean age was 57.11 (SD 15.13), and predominantly male patients (70.27%). The mean age was comparable between the groups (p = 0.676). There was no effect of gender on the vaccination status (p = 0.852). The unvaccinated group had a significantly higher proportion of diabetes mellitus, hypertension, and chronic kidney disease as compared to the vaccinated group (p < 0.001). The APACHE II and SOFA scores were significantly higher in the unvaccinated group (p < 0.01). The invasive ventilation support requirement was significantly higher in the unvaccinated group (45.58%) compared to the vaccinated group (26.44%) (p < 0.001). The mean ABG pH was significantly lower in unvaccinated patients (p < 0.001). The PF ratio was similar between the vaccinated and unvaccinated groups. However, the use of steroids was significantly higher in unvaccinated patients. The median PF ratio was significantly lower in patients on steroids [108 (77–175)] as compared to patients not on steroids [160 (94, 268)] (p < 0.01).

Primary outcome

The ICU mortality was 56.60% (95% CI 53.24–60%). The proportion of mortality was significantly lower among vaccinated patients than unvaccinated patients (43.7% Vs 60.0%, p < 0.0001). The results of multivariable logistic regression and propensity score matching are presented in Table 2.

Table 2 Comparison of the results of MLR and PSM analysis for ICU mortality

The results of logistic regression showed that vaccinated patients had significantly lesser odds of mortality [adjusted odds ratio (AOR), 95% CI 0.52 (0.29, 0.94), p = 0.03] compared to the unvaccinated group adjusted for age, gender, diabetes mellitus, hypertension, APACHE II score, invasive ventilation, ABG pH, need for renal replacement therapy (RRT), and use of steroids (Table 2). The MLR showed higher odds of ICU mortality for age (AOR 1.02, 95% CI 1.004–1.03, p = 0.010), APACHE II score (AOR 1.07, 95% CI 1.03–1.10, p < 0.001), need for invasive ventilation (AOR 2.91, 95% CI 1.95–4.35, p < 0.001), and use of steroids (AOR 2.02, 95% CI 1.14–3.58, p = 0.015). Estimation of vaccination effect by ATE method of PSM analysis also showed that vaccinated patients had significantly lesser odds of mortality compared to the unvaccinated patients (AOR, 95% CI using PSM: 0.83 (0.77, 0.91), p < 0.001). Figure 1 depicts the results of the MLR and PSM using a forest plot explaining the pictorial representation of decreased odds of ICU mortality among vaccinated patients.

Fig. 1

Forest plot of the results MLR and PSM showing adjusted OR with 95% C.I for the ICU mortality. Reported values are adjusted odds ratio (AOR), AOR below 1 indicates the decreased odds of mortality, and AOR above 1 indicates the increased odds of mortality. Vertical line indicates OR of 1- no significant difference between vaccinated and unvaccinated group. ♦ indicates adjusted odds ratio for each covariate. Horizontal line indicates lower and upper 95% confidence interval for the adjusted odds ratio for each covariate

While comparing the estimations of the PSM results with the conventional logistic regression method, the PSM had a lower standard error with a narrower confidence interval than MLR, indicating the estimation of the average treatment effect using PSM was robust. The probability density scores of the vaccinated and the unvaccinated groups before and after the PSM matching for baseline covariates using Kernel density plots are presented in Fig. 2. The Kernel density plots were similar between the vaccinated and unvaccinated groups after matching for baseline covariates using PSM, indicating a strong balance on the propensity scores among the matched samples.

Fig. 2

Kernel density plot of the propensity scores before and after matching. kdensity_pscore—Kernel density propensity scores

Secondary outcomes

Secondary outcomes presented in Table 1. The overall median duration of mechanical ventilation and length of ICU stay were 7 days and were comparable between the vaccinated and unvaccinated groups.

Discussion

This study showed vaccination reduced ICU mortality. The estimate of the effect of vaccination by PSM had a lower standard error and a narrower confidence interval than shown by MLR. The vaccinated patients had less severe disease as shown by lower median APACHE II score, normal ABG pH, less requirement of invasive ventilator support, and steroids. There was no difference in secondary outcomes among vaccinated and unvaccinated patients.

A study from Calabria, Italy, showed lower mortality in vaccinated patients than unvaccinated patients (24.3% vs. 38.5%, p = 0.014) [6]. The vaccinated patients were elderly with a mean age of 67 years (SD 11) and had a significantly higher proportion of comorbidities such as chronic renal failure, autoimmune diseases, and malignancy. Vaccinated patients had a higher PF ratio and lower requirement of invasive ventilator support, but there was no difference in ABG pH among vaccinated and unvaccinated groups. The current study showed no difference in the mean age among vaccinated and unvaccinated groups. The unvaccinated group had higher APACHE II score, increased requirement for invasive ventilator support, steroids, and had a lower ABG pH. This indicates more severe disease in the unvaccinated patients causing alteration in normal physiology and lower ABG pH. The mortality was significantly higher in unvaccinated patients than vaccinated patients (60% vs 43.7%, p < 0.0001). Similar mortality was observed in a study from Turkey [7].

A Spanish study showed vaccinated patients had higher comorbidities and higher APACHE II score 12 (9–17) than unvaccinated patients. There was no difference in the ICU mortality among vaccinated (33.3%) and unvaccinated (28.6%) patients (p = 0.52) [11]. The unvaccinated patients had a higher length of ICU stay than vaccinated patients, but the duration of invasive mechanical ventilation was not different. The vaccinated patients in our study had lower comorbid illnesses and lower median APCHE II score of 14 (8–24.5) than unvaccinated patients. In comparison with the Spanish study, the median APACHE II score in vaccinated patients was higher in our study. We did not find any difference in the length of ICU stay and duration of mechanical ventilation in vaccinated and unvaccinated groups.

The cross-sectional study from Australia by Madeleine Otto, et al. describes the characteristics of vaccinated and unvaccinated patients. The vaccinated patients in the Australian study had higher ICU mortality than unvaccinated patients (18.3% vs 14, p < 0.005). The difference in mortality could be due to elderly population with median age of 64 (51–73) years. The reason stated by the author was the immune escape phenomenon resulting in reduced production of anti-spike antibodies. However, adjusted analysis did not show any difference in mortality between the vaccinated and unvaccinated groups [12].

French study evaluating the impact of vaccination on the severity of illness also showed that the vaccinated cohort was older than the unvaccinated (75 vs.55 yrs) and had a higher proportion of comorbid diseases than the unvaccinated cohort [20]. Although the intensive care unit admissions were lesser in the vaccinated group, the mortality was comparable between the vaccinated and unvaccinated patients in the French study [20]. A study by Grasselli et al., also showed no difference in mortality among vaccinated and unvaccinated patients [9].

Suleyman et al., showed vaccinated patients with breakthrough infection had reduced mortality (11%) as compared to unvaccinated patients (24.9%) (p < 0.001), although the severity of illness was similar between study participants based on the modified SOFA score [21]. Lower mortality observed in the study could be due to inclusion of all the patients requiring hospitalisation. However, in our study the effect of vaccination was estimated among ICU population. We noted significantly higher median SOFA score 7 (4–11) in the unvaccinated patients (p = 0.005).

The strength of this study is this is the first study which compared the mortality in vaccinated and unvaccinated ICU patients by matching the baseline covariates. We suggest using propensity matching to compare the effect of vaccination on ICU mortality. The important observation is vaccination can help in reducing mortality even in the critically ill population.

The limitation of this study is the information on anti-spike antibody titres in the vaccinated group was unavailable. Hence, we could not study the effect of antibody titres on mortality. Also, data regarding genome sequencing to determine the strain causing the infection were unavailable. There is a lot of heterogeneity observed in various studies describing breakthrough infections. One reason could be that different types of vaccines with varying effectiveness could have affected mortality. Also, the time from vaccination can influence the development of breakthrough infections and mortality. The early breakthrough infection could be due to inadequate immune response, and delayed infection could be due to waning immunity [22]. Although vaccination has shown a reduction in mortality in our study, the response of each patient to control COVID-19 infection will be variable. Future research should focus on studying the host immune response to a particular infection as it is one of the parameters in the epidemiological triad.

Conclusion- Estimation of the effect of vaccination by PSM showed lower mortality in the vaccinated COVID-19 patients admitted to ICU. Older age, higher APACHE II score, need for invasive ventilation, and use of steroids had higher odds for ICU mortality. Vaccination is one of the effective tools in controlling the pandemic and has important public health implications.