Abstract

Background: The objective of this study is to validate the admission Glasgow coma scale (GCS) associated with pupil response (GCS-P) to predict traumatic brain injury (TBI) patient’s outcomes in a low- to middle-income country and to compare its performance with that of a simplified model combining the better motor response of the GCS and the pupilar response (MS-P).

Methods: This is a prospective cohort of patients with TBI in a tertiary trauma reference center in Brazil. Predictive values of the GCS, GCS-P, and MS-P were evaluated and compared for 14 day and in-hospital mortality outcomes and length of hospital stay (LHS).

Results: The study enrolled 447 patients. MS-P demonstrated better discriminative ability than GCS to predict mortality (AUC 0.736 × 0.658; P P = 0.073). For hospital mortality, MS-P demonstrated better discrimination than GCS (AUC, 0.750 × 0.682; P P = 0.027). Both scores were good predictors of LHS (r² = 0.084 [GCS-P] × 0.079 [GCS] × 0.072 [MS-P]).

Conclusion: The predictive value of the GCS, GCS-P, and MS-P scales was demonstrated, thus contributing to its external validation in low- to middle-income country.

Keywords: External validation, Glasgow coma scale, Motor score, Prognostic factor, Traumatic brain injury

INTRODUCTION

Traumatic brain injury (TBI) is attributed to a cranial injury that generates a structural or physiological change in the brain. TBI is one of the main causes of mortality and morbidity, with a worldwide incidence between 27 and 69 million cases/year. Outcomes from this disease are not homogeneous when comparing countries with low and high human development index.[ 13 ] The number of TBI cases in low-income countries is increasing due to a higher number of motor vehicles and their related accidents. For high-income countries, better health-care conditions increased life expectancy and, consequently, a higher number of TBI cases from standing height fall in the elderly population.[ 36 ] Brazil is a low- to middle-income country and has experienced an increase in TBI due to both scenarios described. Brazil has more than a million TBI patients, of whom 20–30% have moderate or severe injuries.[ 3 ] The Health Informatics Department (DATASUS) provides data for all diseases in Brazil and has reported an increase of more than 10% in the number of hospitalizations. There are approximately 100,000/year of hospitalizations in Brazil due to TBI.[ 8 , 11 , 39 , 45 ]

The importance of prognosis data is essential for resource allocation, clinical decision-making, and adequate communication for healthcare professionals, patients, and families. This is critical in severe cases, such as TBI. The Glasgow Coma Scale (GCS) is a classic clinical tool used to classify TBI severity. It is associated with outcome and has been used in several prediction models.[ 14 , 17 , 18 , 43 ] The motor component of the GCS appears to be the most powerful predictor of outcome, because it accounts for most of the predictive value of the score and is a component with high intra- and interobserver concordance.[ 17 , 29 ] Recently, Brennan et al. suggested a new model by adding the pupil reactivity score to the GCS, creating the GCS-Pupils (GCS-P).[ 5 ]

Although the authors demonstrated the usefulness of this as a prognostic tool for patients with TBI, this validation was developed according to the international mission for prognosis and clinical trials in TBI (IMPACT)[ 23 ] database, which is a secondary evaluation of multicentric studies from the 1980s and early 1990s, and the corticosteroid randomization after significant head injury[ 33 ] study, which included patients between 1999 and 2005. A study in low- to middle-income countries has shown that these databases overestimate poor neurological outcomes compared to real-world results in patients with severe TBI who underwent early decompressive craniectomy (DC) for damage control.[ 6 ]

Taking into account the specific epidemiological characteristics of TBI in low- to middle-income countries (LMICs),[ 13 , 45 ] as well as the fact that neurosurgeons use DC almost daily for the neurocritical care of patients in resource-limited intensive care unit (ICU),[ 6 , 9 ] this study’s objective was to validate the GCS associated with pupil response (GCS-P) to predict TBI patients outcomes in a LMIC with a recent database and to compare its performance with that of a simplified model combining the better motor response of the GCS and the pupilar response (MS-P).

MATERIALS AND METHODS

Study design, patients, and population

This prospective study was conducted at the Hospital das Clínicas da Faculdade de Medicina de São Paulo (HCFMUSP), one of the largest hospitals in Latin America. Consecutive patients from the emergency department from 2012 to 2015 with TBI were included in the study. No missing data imputation was performed and only patients with full GCS or pupil data were included in the study. Patients under 14 years of age, with penetrating TBI, or coming from another hospital were excluded from the study. Chronic subdural hematomas were also excluded from the study. This research complied with the guidelines of the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement[ 7 ] and the Declaration of Helsinki and was approved by the Research Ethics Committee of the HCFMUSP (Registry: 46831315300000068). Written consent was obtained by all participants or their legal representatives. None of the patients are identifiable in this research.

GCS-P and MS-P

The GCS-P score was calculated by subtracting 0, 1, or 2 points of the GCS when the pupils were bilaterally responsive, only one responsive, and bilaterally irresponsive, respectively (range, 1–15 points). The MS-P was calculated similarly, subtracting the pupillary evaluation of the highest motor response evaluated using the GCS, with a variation of −1–6 points. Scores were calculated based on hospital admission and the clinical picture was evaluated by the emergency team.

Outcome and variables of interest

Patients were followed throughout the hospital stay. The primary outcomes were short-term mortality, considered 14-day mortality, according to the literature used to evaluate acute outcomes of TBI.[ 31 , 35 ] The secondary outcomes were inhospital mortality and length of hospital stay (LHS).

Statistical analysis

Data are expressed as median (quartiles), absolute and relative frequencies, or mean and standard deviation (SD). For categorical variables, the Chi-square test was used. The predictive ability of GCS, GCS-P, and MS-P and its sub-scores to predict mortality outcomes was performed with logistical regression. The discriminative capacity of each logistic model was evaluated by calculating the area under the receiver operating characteristic (ROC) curve (AUC). An AUC or C statistic <0.60 reflects poor discrimination, 0.60–0.75, possibly helpful discrimination, and >0.75, clearly useful discrimination.[ 1 ] The cutoff for each score was chosen as the point that maximized the Youden index (sensitivity + specificity – 1).[ 31 ] To compare AUC, the nonparametric technique described by De Long et al. was used.[ 12 ] The calibration was evaluated using the Hosmer–Lemeshow test. P < 0.05 in the Hosmer–Lemeshow Chi-square test suggests poor calibration.[ 19 ] Overall performance was evaluated using the Brier score, ranging from 0 to 1.[ 16 ] A lower score indicates better model calibration. Spiegelhalter’s z statistic was used to test the significance of Brier scores; a significant result indicates poor calibration.[ 32 ] Discrimination, calibration, and Brier score were reported for the selected predictor variables of interest (GCS, GCS-P, and MS-P) as well as for a multivariate model that included sex, age, and computed tomography (CT)-based Marshall Classification of TBI in addition to the variables mentioned. These variables were selected on the assumption of the literature to be associated with the clinical outcome of patients with TB, which is part of the IMPACT score.[ 23 ]

Data were analyzed using STATA (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC.). ROC curves were performed using MedCalc version 19.1.3 (MedCalc Software bv, Ostend, Belgium; https://www.medcalc.org ; 2020). All hypotheses were constructed as two-tailed, and P < 0.05 within a 95% CI was considered statistically significant.

RESULTS

Patient characteristics

There were a total of 447 patients enrolled. The mean (± SD) age was 40.0 ± 17.8 years (range, 13–99 years), 85.5% were male, and 64.2% were classified as severe TBI. Regarding outcome, 22.8% of the patients died in 14 days and 33.8% in follow-up time. The mean LHS was 25.7 ± 29.7 days and ranged from 1 to 232. The patient characteristics are presented in Table 1 . The median GCS score was 6 (IQR 3–12), the median GCS-P score was 6 (IQR 3–12), and the median MS-P score was 5 (IQR 3–6). Baseline patient characteristics are summarized in Figure 1 .

Table 1:

Patients characteristics for 14-day mortality outcome.

Figure 1:

Percentage of patients in each category of Glasgow coma scale (a), Glasgow coma scale-pupils (b), and motor score-pupils (c) and your sub-scores (d).

Prediction of 14-day mortality

The univariate logistic regression model shows that each GCS score increase was associated with a 12.9% reduction in the mortality risk at 14-days (OR 0.871 [95% CI 0.819–0.926]; P < 0.001), while in the GCS-P, each point on the scale was associated with a reduction of 15.3% (OR 0.847 [95% CI 0.769–0.900]; P < 0.001). Each point increase in MS-P was associated with a 37.4% reduction in 14-day mortality (OR 0.626 [95% CI 0.557–0.703]; P < 0.001). Table 2 shows the discrimination, calibration, and overall performance values for both univariate and multivariate models of all scales. The isolated motor score presented an AUC equal to 0.698, the addition of the pupillary response (MS-P) demonstrated a significant increase in its discriminative ability with an AUC equal to 0.736 (P < 0.001), Figure 2a shows the ROC curves for the MS-P and its components. Univariate analysis of the isolated scales showed that MS-P demonstrated better discrimination than GCS (AUC, 0.736 versus [vs.] 0.658, respectively; P < 0.001) and a greater AUC than the GCS-P; however, the difference was not statistically significant for this outcome (0.736 vs. 0.704; P = 0.073) [ Figure 2b ]. The ideal cutoff point for the GCS score to predict 14-day mortality was 4 (sensitivity, 57.84%; and specificity, 70.14%), five for GCS-P (sensitivity 66.67%; and specificity 66.09%), and three for MS-P (sensitivity 55.88%; and specificity 82.03%). Regarding the calibration and overall performance, all models were adequate. Furthermore, multivariate analysis showed an increase in AUC on all scales; for the 14-day mortality outcome, the highest AUC was obtained for MS-P (AUC = 0.790).

Table 2:

Discriminative ability, calibration and overall performance of all scales for predicting 14-day mortality and intra-hospital mortality.

Figure 2:

Receiver operating characteristics curve for prediction of 14-day mortality by (a) motor score – pupil (MP-P) and its components, (b) Glasgow coma scale (GCS), Glasgow coma scale – pupil (GCS-P) and MS-P.

Prediction of in-hospital mortality

The univariate logistic regression model shows that each GCS score increase was associated with a 14.7% reduction in the odds of in-hospital mortality (OR 0.853 [95% CI 0.808–0.900]; P < 0.001), while in the GCS-P, each point on the scale was associated with a 16.2% reduction in the odds of in-hospital mortality (OR 0.838 [95% CI 0.795–0.884]; P < 0.001). Each point increase in MS-P was associated with a 39.7% reduction in the odds of mortality (OR 0.603 [95% CI 0.537–0.678]; P < 0.001).

The isolated Motor score presented AUC equal to 0.722, the addition of the pupillary response (MS-P) demonstrated a significant increase in its discriminative ability with AUC equal to 0.750 (P = 0.001). Figure 3a shows the ROC curves of the MS-P and its components. MS-P demonstrated better discrimination than GCS (AUC, 0.750 vs. 0.682, respectively; P < 0.001) and a higher AUC than GCS-P, with a statistically significant difference (AUC 0.750 vs. 0.714; P = 0.027) [ Figure 3b ]. The ideal cutoff point for GCS to predict hospital mortality was 3 (sensitivity, 54.30% and specificity, 77.03%), five for GCS-P (sensitivity, 64.24% and specificity, 70.27%), and four for MS-P (sensitivity, 66.89% and specificity, 72.30%). Table 3 shows the comparison of the AUC of the scales obtained in the univariate analysis using the nonparametric model of De Long et al.[ 12 ] The tested models demonstrated adequate calibration and overall performance. Multivariate analysis showed an increase in discriminatory capacity of all scales, with MS-P associated with sex, age, and Marshall CT class presenting a higher AUC compared to the other scales (AUC = 0.813).

Figure 3:

Receiver operating characteristics curve for prediction of in-hospital mortality by (a) motor score – pupil (MP-P) and its components, (b) Glasgow coma scale (GCS), Glasgow coma scale – pupil (GCS-P) and MS-P.

Prediction of LHS

There were a total of 296 patients discharged alive that were included in the LHS prediction analysis. LHS was logarithmically transformed. Linear regression revealed that all scores were moderate predictors of LHS. The GCS-P showed a regression coefficient (β) of −0.289 (95% CI −0.39–−0.18; P < 0.05), compared to −0.280 (95% CI −0.39–−0.17; P < 0.05) for the GCS and −0.268 (95% CI −0.10–−0.44; P < 0.05) for the MS-P. The determination coefficients (R²) were 0.084, 0.079, and 0.072 for the GCS-P, GCS, and MS-P, respectively, thus demonstrating that the GCS-P explained a slightly greater variation in LHS in relation to the other scales. Details of the log LHS analysis for each of the scales are provided in Table 4 .

DISCUSSION

The present study compared the predictive accuracy of GCS and GCS-P in a low- to middle-income country, which is of utmost importance for its validation of TBI in different socioeconomic contexts around the world.[ 4 , 10 , 34 ] In addition, the present study described a new score, MS-P, which is more accurate in assessing 14-day and in-hospital mortality.

Accurate data for prognosis are important for determining appropriate health-care management in life-threatening cases and for proper communication with the patient.[ 1 ] For patients with TBI, effective measures have a substantial potential to change patient outcomes (s).[ 34 , 37 ] It is crucial to employ straightforward prognostic models that can be used in the clinical environment and include relevant information regarding the outcome,[ 15 , 25 ] both to justify decision-making and to facilitate risk communication in a manner that is easily understandable for everyone involved in patient care.[ 24 , 27 , 30 , 37 ]

Solla et al.[ 40 ] suggested, regarding the work of Brennan et al. and Murray et al., who described GCS-P, to report statistical analyses of scale discrimination such as AUC. In the present study, these data were described and corroborated the applicability and utility of GCS-P.[ 5 , 26 , 40 ] It is important to mention that there were some differences regarding the characteristics of the population of their studies, including the mean age.

The three scales demonstrated useful discriminative ability for 14-day and in-hospital mortality, considering that AUC was between 0.60 and 0.75, with emphasis on the MS-P score with the greatest AUC for these two outcomes (0.736 and 0.750, respectively). In addition to having good discrimination, the models presented adequate calibration.

It is not possible to make accurate comparisons regarding the precision of GCS-P in the primary population studied and in the sample of this study, because OR and AUC were not described in the studies mentioned above.[ 5 , 26 ] Nevertheless, it should be noted that differences between the populations studied, the combined data from the Corticosteroid Randomization after Significant Head Injury trial, and IMPACT studies reported mortality of 23.8%, while our study yielded an in-hospital mortality rate of 33.8%. Furthermore, the number of individuals with severe TBI was numerically higher in our population (64.2%) compared to 57% in the previous studies. This reflects the change in the epidemiology of TBI in high-income nations, where falls and older age are now the main contributors to TBI.[ 35 , 36 ] Another important difference is that pupil reactivity’s were determined at different time periods in the IMPACT study, while, in this study, it was obtained at admission.

Table 3:

Comparison between the AUC of the scales.

Table 4:

Simple linear regression analyses in predicting log length of hospital stay by GCS-P score, GCS score and MS-P score.

The higher accuracy of the GCS-P in predicting 14-day and in-hospital mortality compared to the GCS is expected and can be explained as follows. First, pupil reactivity (an important component in the assessment of brainstem reflexes) has been demonstrated in several studies as a strong predictor of TBI-related mortality.[ 18 , 22 , 29 , 41 , 44 ] Second, patients with GCS scores of 3 can be better stratified using the GCS-P, allowing a more accurate prediction of patient outcomes [ Figure 1 ].

LHS is an important epidemiological data point, both to measure health care and to estimate the costs resulting from hospitalization for each pathology.[ 2 , 20 ] Although patient admission assessment scores are commonly used to predict mortality and unfavorable outcomes after TBI, reports in the literature have demonstrated an association between these and patient length of stay in the ICU, as in the study by Okasha et al.[ 29 ] This study is significant in that the scores described the impact on LHS, information that can be useful for managers and clinical practice.

Healey et al.[ 17 ] demonstrated that motor score alone is a better predictor of mortality in relation to GCS since MS contains virtually all information about GCS itself and can be performed in intubated patients. In addition, MS has a linear relationship with mortality, while GCS has a nonlinear function. The higher accuracy of the MS in relation to the other components of the scale was also demonstrated in several other studies.[ 17 , 18 , 20 , 28 , 29 , 43 ] Thus, it is reasonable to use the best motor response associated with pupillary reactivity (i.e., MS-P), which, according to the AUC analysis in the present study, was superior for 14-day and in-hospital mortality prediction to both the GCS and GCS-P.

The prognosis of patients with TBI depends on several variables, which are considered during risk stratification, decision-making, and resource allocation.[ 30 ] Several studies have shown that age and tomography findings are independent predictors of mortality.[ 2 , 22 , 35 ] A multivariate analysis with sex, age, and Marshall Classification associated with the scales showed increased discrimination and a closer relationship between predicted and observed probabilities, as assessed using the Brier score. Therefore, the usefulness of GCS-P and MS-P was demonstrated both separately and in association with the variables commonly used in practice to stratify the risk in patients with TBI.

Lin et al.[ 21 ] have also published a paper on external validation of the e GCS-pupils score, reaching similar results. In their research, they were able to corroborate that the GCS-P score was superior regarding outcome prediction in comparison to the GCS. Regarding in-hospital mortality, GCS presented an AUC of 0.836, MS of 0.820, and a GCS-P of 0.847, while ours were 0.682, 0.722, and 0.714, respectively. Instead of measuring 14-day mortality, their study opted to evaluate 90-day mortality, with an AUC of 0.766 for GCS, 0.742 for MS, and 0.774 for GCS-P, while our 14-day results were 0.682, 0.722, and 0.714 respectively.

It is important to notice that there were significant epidemiological differences between these studies. The gender distribution in their study was almost 50/50, while our database included 85.5% males and 14.5% females, and their mean age was 65.5 compared to our 40. In this regard, our sample is more similar to the IMPACT database, which had a male proportion of 79% and a mean age of 35.7. Another noticeable difference is that Lin et al. used a database in a high IDH country, while our data were collected in a low to middle IDH country. These differences might explain why their study found significantly higher AUC values across all scores.

Study limitations

This is a single-center study, which may limit generalization to other groups. Nonetheless, it is worth mentioning that it is the largest Brazilian city, with a heterogeneous population. The city where the study was conducted receives a high flow of migration from other states of the country. Furthermore, as a tertiary hospital, high-severity patients with complex systemic trauma are commonly admitted. Nevertheless, our work and results are a step toward filling an important information gap in the literature, as several authors have already reported that the GCS motor score is the best predictor of mortality.

This study was performed in a trauma center with a highly specialized and academic environment. However, this is not the reality of most hospitals in LMICs. For that reason, health-care professionals should be properly trained for performing the necessary neurological tests and be able to obtain reliable data that can be used as described in this article.

Based on these findings, we encourage other researchers from numerous locations worldwide to evaluate and describe the accuracy of the MS associated with pupillary reactivity with respect to its predictive power. Furthermore, we suggest that authors from countries, in which the epidemiology of TBI is different from that of the present study, validate the GCS-P in their populations so that it is demonstrably a better predictor of GCS, is widespread, and routinely used in various centers.

We highlight the limitation of not having information on the sample’s long-term results. The Glasgow outcome scale needs to be used to assess results in future investigations. However, the problem with long-term monitoring of TBI patients is not unique to our study and has been discussed in the literature.[ 2 , 29 , 38 , 42 , 46 ]

CONCLUSION

This study validated the usefulness of GCS-P in an LMIC population, thereby contributing to the process of its external validation. GCS-P had greater precision than GCS in predicting 14-day mortality and in-hospital mortality. Moreover, a new score (MS-P) that combines the best GCS motor response with pupillary reactivity was proposed. MS-P demonstrated clearly useful discrimination and higher AUC value than the GCS-P and the GCS for predicting mortality. Discrimination and calibration were also reported for a multivariate model, which included age, sex, and the Marshall CT classification in addition to validated scales. The multivariate model showed an increase in predictive ability compared to individual scales. Another contribution of this study was the evaluation of the relationship between the described scores and LHS, on which there are few reports in the literature, and is very useful for healthcare managers.

Data availability statement

The datasets generated for this study are available on request to the corresponding author.

Declaration of patient consent

Institutional Review Board (IRB) permission obtained for the study.

Financial support and sponsorship

This research was partially funded by the National Council for Scientific and Technological Development (CNPq), Brazil. DJFS, WSP and AGK are supported by the NIHR Global Health Research Group on Neurotrauma, which was commissioned by the National Institute for Health Research (NIHR) using UK aid from the UK Government (project 16/137/105). The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Conflicts of interest

DJFS reports grants and non-financial support from National Institute for Health Research (NIHR), during the conduct of the study. RLOdA reports grants from the National Council for Scientific and Technological Development (CNPq), Brazil, during the conduct of the study. AGK reports grants and non-financial support from NIHR, grants and non-financial support from the School of Clinical Medicine, University of Cambridge, and grants and non-financial support from Royal College of Surgeons of England, during the conduct of the study. WSP reports grants and non-financial support from NIHR, during the conduct of the study.

Disclaimer

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Journal or its management. The information contained in this article should not be considered to be medical advice; patients should consult their own physicians for advice as to their specific medical needs.

References

1. Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ. Discrimination and calibration of clinical prediction models: Users’ guides to the medical literature. JAMA. 2017. 318: 1377-84

2. Amorim RL, Oliveira LM, Malbouisson LM, Nagumo MM, Simoes M, Miranda L. Prediction of early TBI mortality using a machine learning approach in a LMIC population. Front Neurol. 2020. 10: 1-9

3. Anghinah R, de Amorim RL, Paiva WS, Schmidt MT, Ianof JN. Traumatic brain injury pharmacological treatment: Recommendations. Arq Neuropsiquiatr. 2018. 76: 100-3

4. Bonow RH, Barber J, Temkin NR, Videtta W, Rondina C, Petroni G. The outcome of severe traumatic brain injury in Latin America. World Neurosurg. 2018. 111: e82-90

5. Brennan PM, Murray GD, Teasdale GM. Simplifying the use of prognostic information in traumatic brain injury. Part 1: The GCS-Pupils score: An extended index of clinical severity. J Neurosurg. 2018. 128: 1612-20

6. Charry JD, Tejada JH, Pinzon MA, Tejada WA, Ochoa JD, Falla M. Predicted unfavorable neurologic outcome is overestimated by the Marshall computed tomography score, corticosteroid randomization after significant head injury (CRASH), and International mission for prognosis and analysis of clinical trials in traumatic brain injury (IMPACT) models in patients with severe traumatic brain injury managed with early decompressive craniectomy. World Neurosurg. 2017. 101: 554-8

7. Collins GS, Reitsma H, Altman DG, Moons KG. Transparent reporting of a model for individual prognosis or diagnosis: (TRIPOD): The TRIPOD statement. Br Med J. 2014. 68: 134-43

8. De Almeida CE, De Sousa Filho JL, Dourado JC, Gontijo PA, Dellaretti MA, Costa BS. Traumatic brain injury epidemiology in Brazil. World Neurosurg. 2016. 87: 540-7

9. De Amorim R, de Andrade A, Paiva W, Faleiro R, Monteiro R. Management of diffuse lesions in traumatic brain injury in Brazil. Austin Neurosurg Open Access. 2014. 1: 1011

10. De Silva MJ, Roberts I, Perel P, Edwards P, Kenward MG, Fernandes J. Patient outcome after traumatic brain injury in high-, middle-and low-income countries: Analysis of data on 8927 patients in 46 countries. Int J Epidemiol. 2009. 38: 452-8

11. De Souza MR, Fagundes CF, da Costa Sobrinho OP, Ribeiro DG, Muniz JP, Campos LA. Analysis of anatomical lesions and epidemiological study in violent deaths caused by traumatic brain injury. Braz J Neurosurg. 2019. 29: 87

12. De Long ER, De Long DM, Clarke-pearson DL, De Long ER, Carolina N. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 1988. 44: 837-45

13. Dewan MC, Rattani A, Gupta S, Baticulon RE, Hung YC, Punchak M. Estimating the global incidence of traumatic brain injury. J Neurosurg. 2019. 130: 1080-97

14. Duncan R, Thakore S. Decreased Glasgow coma scale score does not mandate endotracheal intubation in the emergency department. J Emerg Med. 2009. 37: 451-5

15. Escher M, Perneger TV, Chevrolet JC. National questionnaire survey on what influences doctors’ decisions about admission to intensive care. Br Med J. 2004. 329: 425-8

16. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996. 15: 361-87

17. Healey C, Osler TM, Rogers FB, Healey MA, Glance LG, Kilgo PD. Improving the glasgow coma scale score: Motor score alone is a better predictor. J Trauma. 2003. 54: 671-80

18. Hoffmann M, Lefering R, Rueger JM, Kolb JP, Izbicki JR, Ruecker AH. Pupil evaluation in addition to Glasgow coma scale components in prediction of traumatic brain injury and mortality. Br J Surg. 2012. 99: 122-30

19. Huang Y, Li W, Macheret F, Gabriel RA, Ohno-Machado L. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Inform Assoc. 2020. 27: 621-33

20. Hyam JA, Welch CA, Harrison DA, Menon DK. Case mix, outcomes and comparison of risk prediction models for admissions to adult, general and specialist critical care units for head injury: A secondary analysis of the ICNARC Case mix programme database. Crit Care. 2006. 10: 1-11

21. Lin Y, Zhang S, Zhang W, Wang X, Huang L, Luo H. The prediction value of Glasgow coma scale-pupils score in neurocritical patients: A retrospective study. Brain Inj. 2021. 35: 547-53

22. Marmarou A, Lu J, Butcher I, McHugh GS, Murray GD, Steyerberg EW. Prognostic value of the Glasgow coma scale and pupil reactivity in traumatic brain injury assessed pre-hospital and on enrollment: An IMPACT analysis. J Neurotrauma. 2007. 24: 270-80

23. Marmarou A, Lu J, Butcher I, McHugh GS, Mushkudiani NA, Murray GD. IMPACT Database of traumatic brain injury: Design and description. J Neurotrauma. 2007. 24: 239-50

24. Menon DK, Schwab K, Wright DW, Maas AI. Position statement: Definition of traumatic brain injury. Arch Phys Med Rehabil. 2010. 91: 1637-40

25. Moore NA, Brennan PM, Baillie JK. Wide variation and systematic bias in expert clinicians’ perceptions of prognosis following brain injury. Br J Neurosurg. 2013. 27: 340-3

26. Murray GD, Brennan PM, Teasdale GM. Simplifying the use of prognostic information in traumatic brain injury. Part. 2: Graphical presentation of probabilities. J Neurosurg. 2018. 128: 1621-34

27. Mushkudiani NA, Hukkelhoven CW, Hernández AV, Murray GD, Choi SC, Maas AI. A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol. 2008. 61: 331-43

28. Namiki J, Yamazaki M, Funabiki T, Hori S. Inaccuracy and misjudged factors of Glasgow coma scale scores when assessed by inexperienced physicians. Clin Neurol Neurosurg. 2011. 113: 393-8

29. Okasha AS, Fayed AM, Saleh AS. The FOUR score predicts mortality, endotracheal intubation and ICU length of stay after traumatic brain injury. Neurocrit Care. 2014. 21: 496-504

30. Perel P, Edwards P, Wentz R, Roberts I. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak. 2006. 6: 1-10

31. Perkins NJ, Schisterman EF. The Youden index and the optimal cut-point corrected for measurement error. Biometrical J. 2005. 47: 428-41

32. Redelmeier DA, Bloch DA, Hickam DH. Assessing predictive accuracy: How to compare brier scores. J Clin Epidemiol. 1991. 44: 1141-6

33. Roberts I, Yates D, Sandercock P, Farrell B, Wasserberg J, Lomas G. Effect of intravenous corticosteroids on death within 14 days in 10008 adults with clinically significant head injury (MRC CRASH trial): Randomised placebo-controlled trial. Lancet. 2004. 364: 1321-8

34. Rocker G, Cook D, Sjokvist P, Weaver B, Finfer S, McDonald E. Clinician predictions of intensive care unit mortality. Crit Care Med. 2004. 32: 1149-54

35. Roozenbeek B, Chiu Y, Lingsma HF, Gerber LM, Steyerberg EW, Ghajar J. Predicting 14-day mortality after severe traumatic brain injury : Application of the IMPACT models in the brain. J Neurotrauma. 2012. 1312: 1306-12

36. Roozenbeek B, Maas AI, Menon DK. Changing patterns in the epidemiology of traumatic brain injury. Nat Rev Neurol. 2013. 9: 231-6

37. Silverberg ND, Gardner AJ, Brubacher JR, Panenka WJ, Li JJ, Iverson GL. Distributio Systematic review of multivariable prognostic models for mild traumatic brain injury. J Neurotrauma. 2015. 32: 517-26

38. Sitsapesan HA, Lawrence TP, Sweasey C, Wester K. Neurotrauma outside the high-income setting: A review of audit and data-collection strategies. World Neurosurg. 2013. 79: 568-75

39. Solla DJ, Quadros DG, Kolias AG, Clark DJ, Hutchinson PJ, Teixeira MJ. Emergency neurosurgery for traumatic brain injury: The need for a national and international registry study. Rev Assoc Med Bras. 2019. 65: 1035-6

40. Solla DJ, Teixeira MJ, Paiva WS. Simplifying the use of prognostic information in patients with traumatic brain injury. J Neurosurg. 2018. 129: 847

41. Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS. Predicting outcome after traumatic brain injury: Development and international validation of prognostic scores based on admission characteristics. PLoS Med. 2008. 5: 1251-61

42. Teasdale GM, Pettigrew LE, Wilson JT, Murray G, Jennett B. Analyzing outcome of treatment of severe head injury: A review and update on advancing the use of the Glasgow outcome scale. J Neurotrauma. 1998. 15: 587-97

43. Vivien B, Yeguiayan J, Le Manach Y, Bonithon-kopp C, Mirek S, Garrigue D. The motor component does not convey all the mortality prediction capacity of the Glasgow Coma scale in trauma patients. Am J Emerg Med. 2012. 30: 1032-41

44. Wartenberg KE, Hwang DY, Haeusler KG, Muehlschlegel S, Sakowitz OW, Madžar D. Gap analysis regarding prognostication in neurocritical care: A joint statement from the german neurocritical care society and the neurocritical care society. Neurocrit Care. 2019. 31: 231-44

45. Whiffin CJ, Smith BG, Esene IN, Karekezi C, Bashford T, Khan MM. Neurosurgeons’ experiences of conducting and disseminating clinical research in low-and middle-income countries: A qualitative study protocol. BMJ Open. 2020. 10: 1-5

46. Wisborg T, Montshiwa TR, Mock C. Trauma research in low-and middle-income countries is urgently needed to strengthen the chain of survival. Scand J Trauma Resusc Emerg Med. 2011. 19: 62