Skip to main content
Annals of Intensive Care
Download PDF

A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score

Annals of Intensive Care volume 11, Article number: 165 (2021) Cite this article

Abstract

Background

Critical care randomized controlled trials (RCTs) are often published in high-impact journals, whether general journals [the New England Journal of Medicine (NEJM), The Lancet, the Journal of the American Medical Association (JAMA)] or critical care journals [Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), Critical Care Medicine (CCM)]. As rejection occurs in up to 97% of cases, it might be appropriate to assess pre-submission probability of being published. The objective of this study was to develop and internally validate a simplified score predicting whether an ongoing trial stands a chance of being published in high-impact general journals.

Methods

A cohort of critical care RCTs published between 1999 and 2018 in the three highest impact medical journals (NEJM, The Lancet, JAMA) or the three highest impact critical care journals (ICM, AJRCCM, CCM) was split into two samples (derivation cohort, validation cohort) to develop and internally validate the simplified score. Primary outcome was journal of publication assessed as high-impact general journal (NEJM, The Lancet, JAMA) or critical care journal (ICM, AJRCCM, CCM).

Results

A total of 968 critical care RCTs were included in the predictive cohort and split into a derivation cohort (n = 510) and a validation cohort (n = 458). In the derivation cohort, the sample size (P value < 0.001), the number of centers involved (P value = 0.01), mortality as primary outcome (P value = 0.002) or a composite item including mortality as primary outcome (P value = 0.004), and topic [ventilation (P value < 0.001) or miscellaneous (P value < 0.001)] were independent factors predictive of publication in high-impact general journals, compared to high-impact critical care journals. The SCOTI score (Sample size, Centers, Outcome, Topic, and International score) was developed with an area under the ROC curve of 0.84 (95% Confidence Interval, 0.80–0.88) in validation by split sample.

Conclusions

The SCOTI score, developed and validated by split sample, accurately predicts the chances of a critical care RCT being published in high-impact general journals, compared to high-impact critical care journals.

Background

Since the late twentieth century, evidence-based medicine has emerged as a new paradigm to produce knowledge and improve clinical practice [1]. Randomized controlled trials (RCTs) are considered as the pinnacle of evidence for evaluating new and existing interventions in medicine [2], as long as they are appropriately designed, conducted, and reported [3, 4]. They have been further promoted by the development of online databases, reviews, meta-analyses, and education of clinicians and students [5].

Critical care is no exception. Many RCTs have been performed in this field and have often led to publications in the highest impact critical care journals, such as Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), and Critical Care Medicine (CCM) [6,7,8]. Moreover, landmark trials have changed the management of critically ill patients [9,10,11,12]. Many of those studies have been published in the highest impact general journals: The New England Journal of Medicine (NEJM), The Lancet, and the Journal of the American Medical Association (JAMA).

It is the final purpose of medical research to broadcast its results as widely as possible to health care practitioners to most impact the management of patients worldwide [13]. In this context, RCTs performed in the field of critical care aim to be published in the highest impact general journals [2]. As up to 97% of trials are rejected outright or post-review, it might be appropriate to assess pre-submission probability of being published. To our knowledge, no study has yet explored the factors associated with the publication of a critical care RCT in a high-impact general journal.

We designed this cohort of studies to identify independent predictive factors of publication of a critical care RCT in a high-impact general journal (NEJM, Lancet, JAMA) compared to the high-impact critical care journals (ICM, AJRCCM, CCM). The secondary objectives of this study were to develop and validate a simplified score predicting publication of an RCT in a high-impact general journal. We hypothesized that most predictive factors of publication of a critical care RCT are determined before the inclusion of its first patient.

Materials and methods

Study design

We conducted a retrospective cohort of RCTs according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline statement [14]. The cohort of RCTs was gathered as a systematic review of RCTs performed in critically ill patients between 1999 and 2018, following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [15].

Data sources and study selection

The search strategy is detailed in the electronic supplementary material (Additional file 1: eAppendix S1). We screened for relevant RCTs performed: (1) on critically ill patients, (2) that enrolled adults, and (3) were published between 1999 and 2018, (4) in the three highest impact general journals [The New England Journal of Medicine (NEJM), The Lancet and the Journal of the American Medical Association (JAMA)] or the three highest impact critical care journals [Intensive Care Medicine (ICM), American Journal of Respiratory and Critical Care Medicine (AJRCCM) and Critical Care Medicine (CCM)]. RCTs performed on animals, on pediatric patients, on non-critically ill patients, bench studies, and simulation studies were not eligible. The three journals for each category (general journals and critical care journals) were selected according to the values of their mean Impact Factor obtained over 3 years from 2016 to 2018 [16]. We chose to only take into account the three “princeps” general journals, which were continuously available from 1999 to 2018. The “sister” journals of a general journal (such as Lancet Respiratory Medicine or JAMA Network) were not included in our research, as they have not been continuously available over the 20-year period, and their respective impact factors are not in the same range as the “princeps” journals.

Data collection

First, two authors (JP and ADJ) independently screened the studies retrieved by title and then by abstract for exclusion. They assessed the full text of possibly relevant studies for inclusion and exclusion criteria. Disagreement was resolved by discussion and arbitrated, if necessary, by a third author (SJ). Data were then added to an excel database, specifically designed for this review [15]. Journal of publication, year of publication, sample size, numbers of centers involved, country of the first author, number of countries participating, primary endpoint of the RCT, the result of the RCT according to its primary endpoint, the type of intervention tested, and the topic of the RCT were extracted.

Journal of publication was classified either as a high-impact general journal (NEJM, Lancet, JAMA) or as a high-impact critical care journal (ICM, AJRCCM, CCM).

The result of the RCT was classified either as unsignificant, significant for benefit, or significant for harm. We adapted a previously published classification [17] to include equivalence and non-inferiority designs. It was considered unsignificant if the P value was higher than 0.05 for superiority trial, or failed to prove the equivalence or the non-inferiority for equivalence and non-inferiority trials. The result of the RCT was considered significant for benefit if the P value was equal or lesser than 0.05 with a better outcome in the intervention group for superiority trials, or if the equivalence or non-inferiority was reached in equivalence or non-inferiority trials, or if the superiority of the intervention was reached in equivalence or non-inferiority trials. The result of the RCT was considered significant for harm if the P value was equal or lesser than 0.05 with a worse outcome in the intervention group for superiority trials, or if the inferiority was reached in equivalence or non-inferiority trials.

The type of intervention tested assessed whether the RCT evaluated a drug or another type of intervention [18].

Statistical analysis

The study size was determined by the total number of RCTs published in the six journals taken into account over the studied period. The database was then split into two cohorts according to the year of publication, to control time-effect and change of policy in the journal’s editing [19]. The derivation cohort included the RCTs published on even years, and the validation cohort included the RCTs published on odd years. We described the cohorts using means, ranges, and SDs as appropriate for continuous variables and frequencies with proportion for categorical variables. We compared proportions using a χ2 test. We compared ordinal categorical variables using a Kruskal–Wallis rank sum test. There was no missing data, so no missing data imputation technique was used [20].

Logistic regression was used to identify predictive factors for publication in a high-impact general journal in the derivation cohort [19, 20]. Continuous variables were split into multiple categories according to their quartiles. A multivariate model was established to predict publication in a high-impact general journal [19, 21]. All variables were selected (regardless of their P value in the univariate analysis) and a stepwise procedure was used to select the final model, according to their Akaike Information Criteria (AIC) [22]. No time effect was found by entering the year of publication variable in a multivariate model. To establish a simplified score, we gave a score to each of the variables included in the final prediction model in relation to each one’s b parameter (regression coefficient) in that model [19]. The discriminative ability of the score was evaluated in both cohorts with receiver operating characteristic (ROC) curves to estimate the area under the curve (AUC), to internally validate the simplified score [21]. We used the bootstrap to internally validate the simplified score by sampling with replacement for 500 iterations [23]. The calibration of the score was graphically assessed by plotting the observed probability (Kaplan–Meier estimates) against the mean predicted probability within tenths of the predicted probabilities [21]. A probability of publication in a high-impact general journal compared to a high-impact critical care journal according to the simplified score was considered low under 20% and high over 80%. A P value of less than or equal to 0.05 was considered statistically significant.

We used SAS, version 9.4 (SAS Institute), for data analysis. The simplified score was developed using rigorous methodological standards, was internally validated using both split-sample and bootstrap validation, and was reported according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline statement [14]. The sample size of the validation cohort met recommendations for validation studies of prediction tools, namely, a minimum of 100 events and a minimum of 100 nonevents [24].

Results

RCTs population

We identified 18,515 articles using the search strategy. We excluded 1780 citations because of duplications and 15,489 citations on the initial abstract screen, because inclusion criteria were not met. After examination of the full text of the 1246 selected papers, we included 968 RCTs in this study. Figure 1 shows the study selection flowchart. 510 RCTs were included in the derivation cohort, and 458 RCTs were included in the validation cohort. 129 RCTs (25%) in the derivation cohort and 106 RCTs (23%) in the validation cohort were published in a high-impact general journal. Overall, the median sample size was 120 [interquartile range (IQR), 46 to 352], the median number of centers was 1 (IQR, 1 to 11) and the median number of countries was 1 (IQR, 1 to 1). For the analysis, the sample size was split into four categories (< 46, 46–120, 121–352, > 352) the number of centers was split into three categories (1, 2–10, > 10), and the number of countries was dichotomized (national design, international design). Characteristics of the derivation and validation cohorts are described in the electronic supplementary material (Additional file 1: Table S1).

Fig. 1
figure 1

Flow chart of the study

Full size image

Final model development

Univariate and multivariate analyses in the derivation cohort are presented in Table 1. In the derivation cohort, the sample size, the number of centers involved, and mortality or composite including mortality primary outcome and topic (ventilation or miscellaneous) were independent predictive factors significantly associated with publication in a high-impact general journal.

Table 1 Predictive factors of publication in high-impact general journals in the derivation cohort, compared to high-impact critical care journals
Full size table

The final multivariate model was constructed with the 510 RCTs of the derivation cohort and all available data. The main predictors for publication in a high-impact general journal were sample size, the number of centers involved, primary outcome, the RCT topic, and an international design. Results of the multivariate logistic regression are presented in Table 2. The AUC of the model was at 0.91 (95% Confidence Interval (95% CI), 0.88–0.94) (Fig. 2a). The calibration plot for the derivation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Figure S1). There was no collinearity in the model. No time effect was found by entering the year of publication variable in a multivariate model (Additional file 1: Table S2). No significant interactions were found, neither between the outcome and the interpretation result, nor between the number of centers and the sample size (Additional file 1: Table S3).

Table 2 Study outcomes
Full size table
Fig. 2
figure 2

Receiver Operating Characteristic curves and Areas Under the Curve to predict publication in high-impact general journals, compared to high-impact critical care journals: a final model in development cohort. b Final model in validation cohort. c SCOTI score in development cohort. d SCOTI score in validation cohort. AUC: Area under the curve, 95%CI: Confidence Interval at 95%

Full size image

Final model validation

Univariate and multivariate analyses in the validation cohort are presented in the electronic supplementary material (Additional file 1: Table S4). Internal validation of the model with the validation cohort indicated high discrimination with a model AUC at 0.85 (95% CI 0.81–0.88) (Fig. 2b). The calibration plot for the validation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Fig. S2).

After internal validation by bootstrap, the final model AUC was at 0.90 (95% CI 0.87–0.92).

SCOTI score development and validation

The simplified score (SCOTI score for Sample size, Centers, Outcome, Topic, and International score) constructed using the final model is described in Fig. 3a, ranging from 0 to 100. The model AUC was at 0.85 (95% Confidence Interval (95% CI), 0.82–0.89) (Fig. 2c). The calibration plot for the derivation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Fig.7 S3). Distribution of the SCOTI score in the derivation cohort is presented in Additional file 1: Fig. S4.

Fig. 3
figure 3

SCOTI score. a SCOTI score calculation worksheet. SCOTI score: sample size, centers, outcome, topic, and international score. b Probability of publication in high-impact general journals compared to high-impact critical care journals according to the SCOTI score: individual prediction. The plot shows predicted (orange curve) and observed (blue dots) probabilities of publication in high-impact general journals according to the SCOTI score. c Probability of publication in high-impact medical journals compared to high-impact critical care journals according to the SCOTI score: three categories. The plot shows observed probabilities of publication in high-impact general journals according to three groups of SCOTI score: “Low probability of success”, “Intermediate probability of success” and “High probability of success”

Full size image

Internal validation of the SCOTI score on the validation cohort indicated high discrimination with model AUC at 0.84 (95% CI 0.80–0.88) (Fig. 2d). The calibration plot for the validation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Figure S5). After internal validation by bootstrap, the AUC of the simplified score was at 0.85 (95% CI 0.82–0.89). Distribution of the SCOTI score in the validation cohort is presented in Additional file 1: Figure S6.

Figure 3b presents the predicted probabilities versus the observed probability of publication in a high-impact general journal on both cohorts in function of the SCOTI score. Figure 3c presents the observed probability of publication in a high-impact general journal according to three ranges of SCOTI score. A SCOTI score between 0 and 45 results in a low probability of publication in a high-impact general journal (6%). A SCOTI score between 50 and 80 results in an intermediate probability of publication in a high-impact general journal (47%). A SCOTI score above 80 results in a high probability of publication in a high-impact general journal (86%). Additional file 1: Table S6 presents the number of RCTs published by each journal per SCOTI score category. NEJM publishes significatively less RCTs with a SCOTI score between 0 and 45 than The Lancet or The JAMA (P < 0.001, Kruskal–Wallis rank sum test).

A cutoff at 50 was determined by ROC analyses, to allow an excellent negative predictive value and a good positive predictive value (i.e., a probability of publication in a high-impact general journal compared to a high-impact critical care journal over 20%). In the validation cohort, positive and negative predictive values (95% CI) were 49% and 93%, respectively, with a sensitivity of 81% and a specificity of 75%.

Discussion

In this predictive cohort of RCTs performed in the field of critical care over 20 years and published in the three highest impact general journals and the three highest impact critical care journals, sample size, number of centers, the primary endpoint, and the topic of the RCT were independent predictive factors of successful publication in a high-impact general journal in multivariate analysis, compared to a high-impact critical care journal. We designed and internally validated by split-sample and bootstrap a model and a simplified score (SCOTI score for Sample size, Centers, Outcome, Topic, and International score). This study shows that a simple easy-to-use model can predict the probability of an RCT being published in a high-impact general journal.

All of the decisive factors of the predictive model and the SCOTI score are determined at trial conception, before inclusion of the first patient [3, 4]. The result of the study does not seem to influence its chances of being published in a high-impact general journal, leading to think that there are few publication bias when it comes to well-designed RCTs [25]. This means that high-impact general journals have taken into account concerns regarding publication bias raised in the 2000s [25]. It is worth noting that it also suggests that critical care RCTs published in NEJM, The Lancet, or the JAMA are specifically designed to address the high requirements of these journals [18, 26]. High-impact general journals have already disclosed their publication policy as focusing on large, international RCTs with solid endpoints. Indeed, high-impact general journals attach importance to the methodological construction of an RCT: its sample size, its multicentric and international character, as well as an objective and pertinent outcome, assessing mortality either as primary endpoint or as a component of the primary endpoint [18]. These results are consistent with a previous study focusing on manuscripts submitted to the British Medical Journal, The Lancet and Annals of Internal Medicine, that found that manuscripts with higher methodology scores were more likely to be published in those journals [27]. In this paper either, the result of a study was not associated with its acceptance. The impact factors of the critical care journals have substantially improved during the studied period, which has probably modified the quality of the RCTs published in these journals.

It is worth noting that the RCTs included in this study were published before the COVID-19 crisis, which may have altered the topics highlighted by high-impact general journals [28]. However, in the studied period sepsis seems to be equally studied in both critical care and general journals, while ventilation trials more often attract general journals.

Our study is to our knowledge the first to highlight independent predictors of publication in a high-impact general journal in the field of critical care. Moreover, it is to our knowledge the largest database of critical care RCTs published to this date, with 968 RCTs published over 20 years in 6 high-impact journals. In this study, we were able to develop and internally validate by split sample and bootstrap a simplified score with excellent discrimination and calibration qualities (Fig. 2, Additional file 1: Figures S3, S4). We could speculate that the SCOTI score presents three major points of interest for research clinicians and journal editors. First, it could help to optimize the methodology used to design an RCT according to the selected criteria which constitute the SCOTI score (Fig. 3) when these criteria are achievable [29]. Second, the SCOTI score might help to accelerate knowledge transfer by assessing pre-submission probability of acceptance. It is an ethical issue to disclose research results to the medical community as fast as possible after trial completion. Multiple submissions to general journals of an RCT with a low probability of acceptance predicted by the SCOTI score could be limited, to avoid waste of time. Third, the SCOTI score might be useful for authors to better understand how the highest impact general journals assess critical care RCTs for possible publication. However, such a score must not be the only component of a submission process, since an RCT with a low SCOTI score but with a revolutionary concept deserves to be granted funding and wide medical diffusion. Some researchers will probably continue to submit their research to general journals rather than directly submit to critical care journals even if their SCOTI scores were low.

Notwithstanding the fact that our model was developed on critical care RCTs, its consequences might be insightful to researchers of other specialties. An external validation cohort on a different field (such as emergency medicine or cardiology) could be complementary to our analysis. Finally, although the score was built over 20 years, no effect of time was identified, neither as fixed nor random effect.

Our study also has limitations. First, we could not assess the manuscripts rejected by each journal over the period. Since our study hypothesizes that every RCT would have been published in a high-impact general journal if it had been accepted by one of them and that every RCT published in a high-impact critical care journal would have been rejected by high-impact general journals, it would have been an added value to compare both submitted and published RCTs to verify this hypothesis. Second, since there is little to no literature on this subject, variables collected and assessed for the model development were selected according to an expert panel (ADJ, GC, EA, EF, NM, SJ) and data on similar subjects [18, 30, 31]. Noteworthily, we chose to dichotomize the country of the study in United States of America versus other, while other classifications might have been used (such as English-speaking countries versus other). Moreover, we did not study in this paper the potential “human factors” (the number of publications of the first or last author, the endorsement by a large consortium or trial group, gender of authors…) influencing the publication process, since it would have highly increased the risk of collinearity with other variables [32]. Similarly, we did not focus on other methodological aspects, such as the absence of bias or the robustness of the statistical analysis. These human and methodological factors should be specifically evaluated in future works [33]. Third, we chose to include in our study the journals with the highest recent Impact Factors. From 2007 at least, data from Harhay and al. indicate that the 6 studied journals are the journals that publish the largest number of RCTs of their respective categories [18]. Likewise, we did not include in our analysis the sistership journals of the high-impact general journals (such as Lancet Respiratory Medicine), since their impact factors is closer to critical care journals than to high-impact general journals. Moreover, their existence is recent, and they cannot be evaluated from 1999 to 2018. Other methods of journals selection might have led to other journals being included. Fourth, since no linearity hypothesis could be done on any of the continuous variables, we had to categorize those variables. We split the variables into categories according to quartiles, since it is a valid and reproducible method. Fifth, we assessed high-impact general journals as a whole, but they might have different policies from one another. Sixth, comparing the characteristics of the trials accepted in the 6 included journals to journals with lower Impact Factors might have been very interesting, since publishing a paper in one of the 3 highest Impact Factor critical care journals is becoming more and more challenging.

Conclusions

In this predictive cohort of critical care RCTs, the sample size, the number of centers, the primary endpoint, and the topic of an RCT were independent predictive factors of successful publication in a high-impact general journal in multivariate analysis, compared to a high-impact critical care journal. The SCOTI score was developed and internally validated by split sample and bootstrap. It predicts the probability of a critical care RCT being published in a high-impact general journal.

Availability of data and materials

Research data and other material (e.g., study protocol and statistical analysis plan) will be made available to the scientific community, immediately on publication, with as few restrictions as possible. All requests should be submitted to the corresponding author who will review with the other investigators for consideration. A data use agreement will be required before the release of participant data and institutional review board approval as appropriate.

Abbreviations

RCTs:

Randomized controlled trials

ICM:

Intensive Care Medicine

AJRCCM:

The American Journal of Respiratory and Critical Care Medicine

CCM:

Critical Care Medicine

NEJM:

New England Journal of Medicine

JAMA:

Journal of the American Medical Association

AIC:

Akaike Information Criteria

ROC:

Receiver operating characteristic

AUC:

Area under the curve

SCOTI:

Sample size, Centers, Outcome, Topic, and International score

References

  1. Group E-BMW. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268:2420–5.

    Article  Google Scholar 

  2. Gattinoni L, Giomarelli P. Acquiring knowledge in intensive care: merits and pitfalls of randomized controlled trials. Intensive Care Med. 2015;41:1460–4.

    Article  Google Scholar 

  3. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement Jama. 1996;276:637–9.

    CAS  Google Scholar 

  4. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–91.

    Article  CAS  Google Scholar 

  5. Montori VM, Guyatt GH. Progress in evidence-based medicine. JAMA. 2008;300:1814–6.

    Article  CAS  Google Scholar 

  6. Santacruz CA, Pereira AJ, Celis E, Vincent JL. Which multicenter randomized controlled trials in critical care medicine have shown reduced mortality? A systematic review. Crit Care Med. 2019;47:1680–91.

    Article  Google Scholar 

  7. Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A. Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med. 2002;28:1316–23.

    Article  Google Scholar 

  8. Pensier J, de Jong A, Hajjej Z, Molinari N, Carr J, Belafia F, Chanques G, Futier E, Azoulay E, Jaber S. Effect of lung recruitment maneuver on oxygenation, physiological parameters and mortality in acute respiratory distress syndrome patients: a systematic review and meta-analysis. Intensive Care Med. 2019;45:1691–702.

    Article  CAS  Google Scholar 

  9. Brochard L, Mancebo J, Wysocki M, Lofaso F, Conti G, Rauss A, Simonneau G, Benito S, Gasparetto A, Lemaire F, et al. Noninvasive ventilation for acute exacerbations of chronic obstructive pulmonary disease. N Engl J Med. 1995;333:817–22.

    Article  CAS  Google Scholar 

  10. Papazian L, Forel JM, Gacouin A, Penot-Ragon C, Perrin G, Loundou A, Jaber S, Arnal JM, Perez D, Seghboyan JM, Constantin JM, Courant P, Lefrant JY, Guérin C, Prat G, Morange S, Roch A. Neuromuscular blockers in early acute respiratory distress syndrome. N Engl J Med. 2010;363:1107–16.

    Article  CAS  Google Scholar 

  11. Jaber S, Lescot T, Futier E, Paugam-Burtz C, Seguin P, Ferrandiere M, Lasocki S, Mimoz O, Hengy B, Sannini A, Pottecher J, Abback PS, Riu B, Belafia F, Constantin JM, Masseret E, Beaussier M, Verzilli D, De Jong A, Chanques G, Brochard L, Molinari N. Effect of noninvasive ventilation on tracheal reintubation among patients with hypoxemic respiratory failure following abdominal surgery: a randomized clinical trial. JAMA. 2016;315:1345–53.

    Article  CAS  Google Scholar 

  12. Annane D, Sébille V, Charpentier C, Bollaert PE, François B, Korach JM, Capellier G, Cohen Y, Azoulay E, Troché G, Chaumet-Riffaud P, Bellissant E. Effect of treatment with low doses of hydrocortisone and fludrocortisone on mortality in patients with septic shock. JAMA. 2002;288:862–71.

    Article  CAS  Google Scholar 

  13. Bellani G, Laffey JG, Pham T, Fan E, Brochard L, Esteban A, Gattinoni L, van Haren F, Larsson A, McAuley DF, Ranieri M, Rubenfeld G, Thompson BT, Wrigge H, Slutsky AS, Pesenti A. Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 Countries. JAMA. 2016;315:788–800.

    Article  CAS  Google Scholar 

  14. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:7594.

    Article  Google Scholar 

  15. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JP, Straus S, Thorlund K, Jansen JP, Mulrow C, Catalá-López F, Gøtzsche PC, Dickersin K, Boutron I, Altman DG, Moher D. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162:777–84.

    Article  Google Scholar 

  16. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90–3.

    Article  CAS  Google Scholar 

  17. Ospina-Tascón GABG, Vincent JL. Multicenter, randomized, controlled trials evaluating mortality in intensive care: doomed to fail? Crit Care Med. 2008;36:1311–22.

    Article  Google Scholar 

  18. Harhay MO, Wagner J, Ratcliffe SJ, Bronheim RS, Gopal A, Green S, Cooney E, Mikkelsen ME, Kerlin MP, Small DS, Halpern SD. Outcomes and statistical power in adult critical care randomized trials. Am J Respir Crit Care Med. 2014;189:1469–78.

    Article  Google Scholar 

  19. De Jong A, Molinari N, Terzi N, Mongardon N, Arnal JM, Guitton C, Allaouchiche B, Paugam-Burtz C, Constantin JM, Lefrant JY, Leone M, Papazian L, Asehnoune K, Maziers N, Azoulay E, Pradel G, Jung B, Jaber S. Early identification of patients at risk for difficult intubation in the intensive care unit: development and validation of the MACOCHA score in a multicenter cohort study. Am J Respir Crit Care Med. 2013;187:832–9.

    Article  Google Scholar 

  20. Leisman DE, Harhay MO, Lederer DJ, Abramson M, Adjei AA, Bakker J, Ballas ZK, Barreiro E, Bell SC, Bellomo R, Bernstein JA, Branson RD, Brusasco V, Chalmers JD, Chokroverty S, Citerio G, Collop NA, Cooke CR, Crapo JD, Donaldson G, Fitzgerald DA, Grainger E, Hale L, Herth FJ, Kochanek PM, Marks G, Moorman JR, Ost DE, Schatz M, Sheikh A, Smyth AR, Stewart I, Stewart PW, Swenson ER, Szymusiak R, Teboul JL, Vincent JL, Wedzicha JA, Maslove DM. Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals. Crit Care Med. 2020;48:623–33.

    Article  Google Scholar 

  21. Elliott J, Bodinier B, Bond TA, Chadeau-Hyam M, Evangelou E, Moons KGM, Dehghan A, Muller DC, Elliott P, Tzoulaki I. Predictive Accuracy of a Polygenic Risk Score-Enhanced Prediction Model vs a Clinical Risk Score for Coronary Artery Disease. JAMA. 2020;323:636–45.

    Article  Google Scholar 

  22. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:604.

    Article  Google Scholar 

  23. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–81.

    Article  CAS  Google Scholar 

  24. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58:475–83.

    Article  Google Scholar 

  25. Murray SB, Heathers JA, Schauer RM, Griffiths S, Mitchison D, Mond JM, Nagata JM. Postpublication metrics of randomized clinical trials with and without null findings. JAMA. 2019;321:1825–6.

    Article  Google Scholar 

  26. Latronico N, Metelli M, Turin M, Piva S, Rasulo FA, Minelli C. Quality of reporting of randomized controlled trials published in Intensive Care Medicine from 2001 to 2010. Intensive Care Med. 2013;39:1386–95.

    Article  Google Scholar 

  27. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006;184:621–6.

    Article  Google Scholar 

  28. Bauchner H, Fontanarosa PB. Randomized Clinical Trials and COVID-19: Managing Expectations. JAMA. 2020;323:2262–3.

    Article  CAS  Google Scholar 

  29. Harhay MO, Casey JD, Clement M, Collins SP, Gayat É, Gong MN, Jaber S, Laterre PF, Marshall JC, Matthay MA, Monroe RE, Rice TW, Rubin E, Self WH, Mebazaa A. Contemporary strategies to improve clinical trial design for critical care research: insights from the First Critical Care Clinical Trialists Workshop. Intensive Care Med. 2020;46:930–42.

    Article  Google Scholar 

  30. Duffett M, Choong K, Hartling L, Menon K, Thabane L, Cook DJ. Randomized controlled trials in pediatric critical care: a scoping review. Crit Care (London, England). 2013;17:R256.

    Article  Google Scholar 

  31. Gaudry S, Messika J, Ricard JD, Guillo S, Pasquet B, Dubief E, Boukertouta T, Dreyfuss D, Tubach F. Patient-important outcomes in randomized controlled trials in critically ill patients: a systematic review. Ann Intensive Care. 2017;7:28.

    Article  Google Scholar 

  32. Fraser GE, Stram DO. Regression calibration in studies with correlated variables measured with error. Am J Epidemiol. 2001;154:836–44.

    Article  CAS  Google Scholar 

  33. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Nonscientific factors associated with acceptance for publication in The Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008;90:2432–7.

    Article  Google Scholar 

Download references

Acknowledgements

None.

Funding

The funding source had no role in the conception, design, or conduct of the trial, nor did their representatives participate in the collection, management, analysis, interpretation, or presentation of the data or in the preparation, review, or approval of the manuscript.

Author information

Authors and Affiliations

  1. Department of Anesthesia and Intensive Care Unit, Regional University Hospital of Montpellier, St-Eloi Hospital, University of Montpellier, PhyMedExp, INSERM U1046, CNRS UMR, 9214 CEDEX 5, Montpellier, France

    Joris Pensier, Audrey De Jong, Gerald Chanques & Samir Jaber

  2. Department of Peri-Operative Medicine, CHU Clermont-Ferrand, 63000, Clermont-Ferrand, France

    Emmanuel Futier

  3. Médecine Intensive et Réanimation, Groupe FAMIREA, Hôpital Saint-Louis, Université de Paris, Paris, France

    Elie Azoulay

  4. IDESP, INSERM, Univ Montpellier, CHU Montpellier, Montpellier, France

    Nicolas Molinari

  5. Universite de Montpellier, Montpellier, Languedoc-Roussillon, France

    Nicolas Molinari

  6. Département d’Anesthésie Réanimation B (DAR B), 80 Avenue Augustin Fliche, 34295, Montpellier, France

    Samir Jaber

Authors
  1. Joris Pensier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Audrey De Jong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerald Chanques
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emmanuel Futier
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Elie Azoulay
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nicolas Molinari
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Samir Jaber
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JP, ADJ, NM and SJ designed the study. JP and ADJ collected the data. JP, ADJ, NM and SJ analyzed the data. All authors were involved in the data interpretation. JP, ADJ, NM, and SJ wrote the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Samir Jaber.

Ethics declarations

Ethics approval, consent to participate and consent to publication

Not applicable.

Competing interests

Pr. Jaber reports receiving consulting fees from Drager, Medtronic, Baxter, Fresenius-Xenios, and Fisher & Paykel. Pr. Azoulay has received fees for lectures from Gilead, Pfizer, Baxter, and Alexion. His research group has been supported by Ablynx, Fisher & Paykel, Jazz Pharma, and MSD. Dr. De Jong reports receiving consulting fees from Medtronic. No conflict of interests is reported for other authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. Appendix S1

. Search strategy. Table S1. Description of derivation and validation cohorts. Table S2. Multivariate model to assess time effect. Table S3. Multivariate model to assess interaction between outcome and interpretation result and interaction between number of centers and sample size. Table S4. Univariate and multivariate analysis on the development cohort. Table S5. Univariate and multivariate analysis on the validation cohort. Table S6. Number of RCTs per SCOTI score category per journal. Figure S1. Calibration plot for the final model on the derivation cohort. Figure S2. Calibration plot for the final model on the validation cohort. Figure S3. Calibration plot for the SCOTI score on the derivation cohort. Figure S4. Distribution of the SCOTI score in the derivation cohort. Figure S5. Calibration plot for the SCOTI score on the validation cohort. Figure S6. Distribution of the SCOTI score in the validation cohort.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pensier, J., De Jong, A., Chanques, G. et al. A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score. Ann. Intensive Care 11, 165 (2021). https://doi.org/10.1186/s13613-021-00954-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13613-021-00954-x

Keywords