Easily created prediction model using deep learning software (Prediction One, Sony Network Communications Inc.) for subarachnoid hemorrhage outcomes from small dataset at admission
- Department of Neurosurgery, Suwa Red Cross Hospital, Suwa, Nagano, Japan.
DOI:10.25259/SNI_636_2020Copyright: © 2020 Surgical Neurology International This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-Share Alike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as the author is credited and the new creations are licensed under the identical terms.
How to cite this article: Masahito Katsuki, Yukinari Kakizawa, Akihiro Nishikawa, Yasunaga Yamamoto, Toshiya Uchiyama. Easily created prediction model using deep learning software (Prediction One, Sony Network Communications Inc.) for subarachnoid hemorrhage outcomes from small dataset at admission. 06-Nov-2020;11:374
How to cite this URL: Masahito Katsuki, Yukinari Kakizawa, Akihiro Nishikawa, Yasunaga Yamamoto, Toshiya Uchiyama. Easily created prediction model using deep learning software (Prediction One, Sony Network Communications Inc.) for subarachnoid hemorrhage outcomes from small dataset at admission. 06-Nov-2020;11:374. Available from: https://surgicalneurologyint.com/surgicalint-articles/10375/
Background: Reliable prediction models of subarachnoid hemorrhage (SAH) outcomes are needed for decision-making of the treatment. SAFIRE score using only four variables is a good prediction scoring system. However, making such prediction models needs a large number of samples and time-consuming statistical analysis. Deep learning (DL), one of the artificial intelligence, is attractive, but there were no reports on prediction models for SAH outcomes using DL. We herein made a prediction model using DL software, Prediction One (Sony Network Communications Inc., Tokyo, Japan) and compared it to SAFIRE score.
Methods: We used 153 consecutive aneurysmal SAH patients data in our hospital between 2012 and 2019. Modified Rankin Scale (mRS) 0–3 at 6 months was defined as a favorable outcome. We randomly divided them into 102 patients training dataset and 51 patients external validation dataset. Prediction one made the prediction model using the training dataset with internal cross-validation. We used both the created model and SAFIRE score to predict the outcomes using the external validation set. The areas under the curve (AUCs) were compared.
Results: The model made by Prediction One using 28 variables had AUC of 0.848, and its AUC for the validation dataset was 0.953 (95%CI 0.900–1.000). AUCs calculated using SAFIRE score were 0.875 for the training dataset and 0.960 for the validation dataset, respectively.
Conclusion: We easily and quickly made prediction models using Prediction One, even with a small single-center dataset. The accuracy of the model was not so inferior to those of previous statistically calculated prediction models.
Keywords: Artificial intelligence, Deep learning, Machine learning, Prediction model, Subarachnoid hemorrhage
Subarachnoid hemorrhage (SAH) due to the ruptured cerebral aneurysm is one of the severe stroke types, and it has a mortality rate of up to 35%. Furthermore, about one-third of the patients remain functionally dependent.[
According to the Japanese Guidelines for the Management of Stroke 2015,[
Previously, many studies tried to make the prediction model for SAH outcomes,[
Similar to these studies, statistically making a prediction model or scoring system need a large number of samples over thousands, so these studies tend to be country-initiated or academic association-initiated research. However, the larger the sample size, the less detailed information is available, such as comorbidities, use of antithrombotic drugs, or laboratory test data and the more there are missing data. Furthermore, the treatment strategies vary from hospital to hospital, and patient backgrounds differ depending on countries and regions. Therefore, these prediction models work as the greatest common denominator worldwide, but not necessarily applicable, with high accuracies, to the respective hospital.
Recently, artificial intelligence (AI) is attracting. Especially, it is a transitional period regarding AI from machine learning to deep learning (DL). Machine learning, such as random forest, logistic regression, or clustering, is defined as “Algorithms that parse data, learn from that data, and then apply what they have learned to make informed decisions.” On the other hand, DL is considered an evolution of machine learning. DL uses a programmable neural network that enables machines to make accurate decisions without help from humans. To achieve this, DL applications use a layered structure of algorithms called an artificial neural network (ANN). The design of ANN is inspired by the biological neural network of the human brain, leading to a process of learning that’s far more capable than that of standard machine learning models.
Machine learning has been used in neurosurgical situations,[
We retrospectively retrieved data from medical records of all the consecutive 153 aneurysmal SAH patients who were admitted between 2012 and 2019 and treated at our institution. Patients with cardiopulmonary arrest on arrival were excluded from the study. The diagnosis of SAH was based on the clinical history and the presence of SAH on CT. The hospital’s research ethics committee approved this study, and we gained written informed consent for this study from all of the patients, the legally authorized representative of the patients, or next of kin of the deceased patients. All methods were carried out in accordance with relevant guidelines and regulations (Declaration of Helsinki).
General management of SAH was similar in all cases: all patients were first treated with nicardipine and kept normovolemic with normal blood pressure (systolic blood pressure <140 mmHg). Indication for surgery was established according to the Japanese Guidelines for the Management of Stroke 2009[
All SAH patients who underwent aneurysm clipping or coiling received fasudil, cilostazol, and statin as appropriate after the operation. Rehabilitation of 150 days as maximum and nutritional support was started as soon as possible after the operation, and prophylaxis and treatment of complications were also ensured. Intra-arterial infusion of fasudil was performed when necessary for the treatment of symptomatic vasospasm. In addition, a ventriculoperitoneal shunt was performed when hydrocephalus was observed.
We collected data regarding physiological symptoms at admission for patients included in this study, that is, age, sex, height, weight, WFNS grade, systolic blood pressure, administration of antithrombotic drugs, history of smoking and drinking, hypertension, diabetes mellitus, and dyslipidemia. We also measured albumin, white blood cell, lymphocyte, triglycerides, total cholesterol, high-density lipoprotein cholesterol, low-density lipoprotein cholesterol, glucose, hemoglobin A1c, and brain natriuretic peptide levels at admission. Albumin, lymphocyte, and total cholesterol are known factors for controlling nutritional status score to assess the nutritional status of the patients.[
We determined the location (anterior cerebral artery, anterior communicating artery, internal carotid artery, middle cerebral artery, posterior cerebral artery, basilar artery, vertebral artery, or undetermined due to severe neurological status treated by conservative therapy as well as diffuse SAH with multiple aneurysms which we could not be judged as ruptured) and size (mm) of the aneurysm, Fisher CT scale,[
We also investigated the treatment strategy (clipping, coiling, or others, including cerebral ventricular drainage or conservative therapy). To evaluate the outcomes, mRS scores at 6 months after the treatment of all 153 patients were collected by either personal outpatient interviews, reports from the rehabilitation hospital or home doctor, or interviews over the telephone, once the ethical approval was obtained for the study. We dichotomized mRS scores into favorable (mRS 0–3) or poor (mRS 4–6).
Making prediction model by Prediction One
We used Prediction One software to make the prediction model. We divided our 153 patients data randomly into 102 patients training dataset and 51 patients external validation dataset. Prediction One read the 102 patients data and automatically divided them into almost half as internal training and cross-validation datasets. Prediction One automatically adjusted and optimized the variables in a way that is easy to process statistically and mathematically, and select appropriate algorithm with ensemble learning. The missing values were automatically compensated and Prediction One made the best prediction model by ANN with internal cross-validation. The details are trade secrets and could not be provided.
We let the Prediction One software make 2 prediction models using 102 patients training dataset. One was made using all 28 variables acquired at admission, and the other used only four variables which are used for SAFIRE score;[
Prediction using SAFIRE score
As the third model in this study, we also investigated SAFIRE scores[
SAFIRE score is a simple scoring system predicting the probability of the outcomes as mRS 0–3 or 4–6 at 2 months. In the original article of the SAFIRE score,[
Results are shown as median (interquartile range). The difference between the training dataset and the external validation dataset was tested using the Mann–Whitney U test, Fisher’s exact test, or Pearson’s Chi-square test. A two-tailed P < 0.05 was considered statistically significant. We calculated AUCs and their P-values using SPSS software version 24.0.0 (IBM, New York, USA).
The clinical characteristics of the 153 SAH patients (100 women and 53 men) are summarized in [
Model development and validation
Prediction One produced each prediction model in <1 min. The AUCs of each model were described in [
The stronger variables of each model are listed in [
Comparison to SAFIRE score
We calculated the SAFIRE score using 95 of the 102 patients in the training dataset and 46 of the 51 patients in the validation datasets, respectively. The AUCs were 0.875 (95% CI 0.807–0.943) and 0.960 (95% CI 0.905–1.000), respectively [
We made prediction models using DL software, Prediction One, and we created models with a high prediction rate using a small dataset with several missing data, and it would be reliable for the prediction in our own hospital. Furthermore, this is the first report on creating a prediction model of SAH using DL.
Advantages of DL
DL is now widely used not only in medicine but also in economics, sociology, and logistics. DL is used by a variety of companies with few variables and data, such as forecasting store stocking, predicting insurance policy renewals, and predicting stock prices and land values. Although the sample size in medical situations may be small compared to them, there are many chronological, cross-sectional, and nonnumeric variables, such as chief complaint, medical history, family history, blood test data, vital signs, sentences in the electronic medical record themselves, and radiological imaging data with detailed radiomics, that may be suitable for creating predictive models with DL.
Conventional time and cost-consuming statistical analysis need variable optimization like a logarithmic transformation for making the variables like the normal distribution to increase the accuracy of the prediction model. It also requires the arbitrary selection of variables based on previous studies and multivariate analysis needs 10 folds number of samples against the variables.[
However, DL has the potential to overcome these problems. In the statistical analysis, we should perform variable optimization and sometimes choose variables arbitrarily due to small sample size. However, DL can develop useful prediction models without those time-consuming or arbitrary procedures because DL software automatically does these processes. Furthermore, the number of the variables for DL software is not limited, and DL sometimes find interesting variables as important that has not been taken into account in the previously reported statistical models.[
We then review these benefits of DL in our study. Conventionally, we could have used only ten variables for statistical analysis due to the small sample size of the training dataset (n = 102). Furthermore, the dataset contains several missing data. However, we could use 28 variables for making a prediction model by Prediction One, and make a good prediction model from the small dataset. We did not need to perform variable optimization nor manipulations for the missing values. Furthermore, some of the serological test results like glucose level were judged to be important among many other previously reported important factors such as aneurysm location and Fisher CT scale. Besides, the time needed for creating each model was <1 min. Finally, the models achieved high accuracy with AUC of 0.848 in the training dataset and 0.953 in the validation dataset. Putting it bluntly, this means our prediction model, even made from the small dataset, can predict the SAH patients’ outcomes treated in our hospital with as high accuracy of SAFIRE score, which was made from the large cohort study.
Limitations of DL
First, the adequate sample size is still unknown for DL. Fujita et al. reported that there were no differences in the accuracy of the predicting independent dressing on month after stroke among 80, 100, and 120 patients dataset. However, the accuracy was worse under 60 patients dataset.[
Second, DL can treat images or sentences, and this is a very strong point compared to other AI algorithms, but we did not use these advantages. We could have used CT images or descriptions themselves in the electronic medical record to making prediction models. Besides, the number of variables for DL is unlimited so, we could have used more variables that are not reported previously such as other laboratory test data, chronological changes of vital signs, meteorological conditions, and ore calendarial factors. Further study or programming should be needed.
Third, the prediction model derived from our own data cannot be applied to other institutions, and the training and validation dataset must be updated to keep up with advances in medical science and changes in surgical techniques. Creating an AI-based prediction model that can be used universally at any hospital will still require country-initiated or academic association-initiated collaborative research at many institutions and may require the same amount of effort as the traditional statistical model creation.
Fourth, we need to examine the clinical usefulness of the prediction model prospectively. For example, by performing surgery only to patients with a good prognosis predicted by the model, we would evaluate the reduction of the workload and stress of the medical staff, and the medical resources and costs could be saved or not. Furthermore, we should examine the changes of the families’ minds in decision making to treat the patients whose outcomes are empirically difficult to predict, such as elderly patients with SAH Grade III, after showing the result of AI prediction.
Fifth, we used Prediction One software, but there are many AI software (frameworks) worldwide, and there are a thousand different ways to assemble the ANN (libraries). Prediction one is suitable for predicting binomial, ordinal, or continuous variables and can treat Japanese sentences themselves. Furthermore, when there are missing values, it automatically compensates. However, the details of how the neural network is assembled and tuned have not been released, so we need to think carefully about the accuracy of the models and why the variables were judged as important, considering the clinical meaning.
Despite this easiness, advantages, and future potential of DL, the majority of medical staff cannot treat DL software. Staartjes et al. reported this is because of lack of skilled resources (staff, equipment) to develop a model, time limitations restricting AI application in clinical practice, lack of AI models for the indications of interest, uncertainty concerning which processes may benefit most from the application of AI algorithms, lack of data to develop a model, and lack of personal convincement of the added value of this new technology.[
For example, in Japan, a nation-wide study revealed that 77% of the 5344 SAH patients underwent clipping and others coiling from 1999 to 2012,[
Furthermore, the big data have been stored, such as coronavirus disease 2019 Public Datasets,[
Limitation of this study
We used WFNS grade at admission, but the SAFIRE score used the WFNS score assessed after neurological resuscitation[
We easily and quickly made prediction models using Prediction One software. The accuracies of the prediction models were not inferior to those of previous statistically calculated prediction models. Even with a small single-center dataset, containing missing data, prognostic models made by DL software can be useful at the institution and may be applied to daily clinical practice in the future.
1. Abulhasan YB, Alabdulraheem N, Simoneau G, Angle MR, Teitelbaum J. Mortality after spontaneous subarachnoid hemorrhage: Causality and validation of a prediction model. World Neurosurg. 2018. 112: e79-811
2. Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S. Use of artificial neural networks to decision making in patients with lumbar spinal canal stenosis. J Neurosurg Sci. 2017. 61: 603-11
3. Donkelaar CE, Bakker NA, Birks J, Veeger NJ, Metzemaekers JD, Molyneux AJ. Prediction of outcome after aneurysmal subarachnoid hemorrhage: Development and validation of the SAFIRE grading scale. Stroke. 2019. 50: 837-44
4. Donkelaar CE, Bakker NA, Veeger NJ, Uyttenboogaart M, Metzemaekers JD, Eshghi O. Prediction of outcome after subarachnoid hemorrhage: Timing of clinical assessment. J Neurosurg. 2017. 126: 52-9
5. Fisher CM, Kistler JP, Davis JM. Relation of cerebral vasospasm to subarachnoid hemorrhage visualized by computerized tomographic scanning. Neurosurgery. 1980. 6: 1-9
6. Fujita T, Ohashi T, Yamane K, Yamamoto Y, Sone T, Ohira Y. Relationship between the number of samples and the accuracy of the prediction model for dressing independence using artificial neural networks in stroke patients. Jpn J Compr Rehabil Sci. 2020. 11: 28-34
7. Fukuma R, Yanagisawa T, Kinoshita M, Shinozaki T, Arita H, Kawaguchi A. Prediction of IDH and TERT promoter mutations in low-grade glioma from magnetic resonance images using a convolutional neural network. Sci Rep. 2019. 9: 20311
8. Furtner J, Berghoff AS, Albtoush OM, Woitek R, Asenbaum U, Prayer D. Survival prediction using temporal muscle thickness measurements on cranial magnetic resonance images in patients with newly diagnosed brain metastases. Eur Radiol. 2017. 27: 3167-73
9. Furtner J, Berghoff AS, Schöpf V, Reumann R, Pascher B, Woitek R. Temporal muscle thickness is an independent prognostic marker in melanoma patients with newly diagnosed brain metastases. J Neurooncol. 2018. 140: 173-8
10. Furtner J, Genbrugge E, Gorlia T, Bendszus M, Nowosielski M, Golfinopoulos V. Temporal muscle thickness is an independent prognostic marker in patients with progressive glioblastoma: Translational imaging analysis of the EORTC 26101 trial. Neuro Oncol. 2019. 21: 1587-94
11. Fuse N, Sakurai-Yageta M, Katsuoka F, Danjoh I, Shimizu R, Tamiya G. Establishment of integrated biobank for precision medicine and personalized healthcare: The Tohoku medical megabank project. JMA J. 2019. 2: 113-22
12. Google Cloud Platform, COVID-19 Public Datasets. Available from: https://www.console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-public-data-program?_ga=2.6492658.-1999449961.1599963188&pli=1 [Last accessed on 2020 Jun 29].
13. Hostettler IC, Sebök M, Ambler G, Muroi C, Prömmel P, Neidert MC. Validation and optimization of barrow neurological institute score in prediction of adverse events and functional outcome after subarachnoid hemorrhage-creation of the HATCH (hemorrhage, age, treatment, clinical state, Hydrocephalus) score. Neurosurgery. 2020. p.
14. Hunt WE, Hess RM. Surgical risk as related to time of intervention in the repair of intracranial aneurysms. J Neurosurg. 1968. 28: 14-20
15. Hunt WE, Kosnik EJ. Timing and perioperative care in intracranial aneurysm surgery. Clin Neurosurg. 1974. 21: 79-89
16. Ido K, Nakamura N, Nakayama M. Miyagi medical and welfare information network: A backup system for patient clinical information after the great east Japan earthquake and tsunami. Tohoku J Exp Med. 2019. 248: 19-25
17. Ignacio de Ulíbarri J, González-Madroño A, de Villar NG, González P, González B, Mancha A. CONUT: A tool for controlling nutritional status. First validation in a hospital population. Nutr Hosp. 2005. 20: 38-45
18. Iihara K, Tominaga T, Saito N, Suzuki M, Date I, Fujii Y. The japan neurosurgical database: Overview and results of the first-year survey. Neurol Med Chir (Tokyo). 2020. 60: 165-90
19. Jaja BN, Cusimano MD, Etminan N, Hanggi D, Hasan D, Ilodigwe D. Clinical prediction models for aneurysmal subarachnoid hemorrhage: A systematic review. Neurocrit Care. 2013. 18: 143-53
20. Jaja BN, Saposnik G, Lingsma HF, Macdonald E, Thorpe KE, Mamdani M. Development and validation of outcome prediction models for aneurysmal subarachnoid haemorrhage: The SAHIT multinational cohort study. BMJ. 2018. 360: j5745
21. Kamada F, Aoki Y, Narisawa A, Abe Y, Komatsuzaki S, Kikuchi A. A genome-wide association study identifies RNF213 as the first Moyamoya disease gene. J Hum Genet. 2011. 56: 34-40
22. Katsuki M, Kakizawa Y, Nishikawa A, Yamamoto Y, Uchiyama T. Lower total protein and absence of neuronavigation are novel poor prognostic factors of endoscopic hematoma removal for intracerebral hemorrhage. J Stroke Cerebrovasc Dis. 2020. 29: 105050
23. Katsuki M, Kakizawa Y, Nishikawa A, Yasunaga Y, Uchiyama T. Endoscopic hematoma removal of supratentorial intracerebral hemorrhage under local anesthesia reduces operative time compared to craniotomy. Sci Rep. 2020. 10: 10389
24. Katsuki M, Suzuki Y, Kunitoki K, Sato Y, Sasaki K, Mashiyama S. Temporal muscle as an indicator of sarcopenia is independently associated with hunt and kosnik grade on admission and the modified rankin scale score at 6 months of patients with subarachnoid hemorrhage treated by endovascular coiling. World Neurosurg. 2020. 137: e526-34
25. Katsuki M, Suzuki Y, Sato Y, Sasaki K, Shingai Y, Mashiyama S. In reply to the letter to the editor regarding temporal muscle as an indicator of sarcopenia is independently associated with hunt and kosnik grade on admission and the modified rankin scale at 6 months of patients with subarachnoid hemorrhage treated by endovascular coiling. World Neurosurg. 2020. 140: 433
26. Katsuki M, Yamamoto Y, Uchiyama T, Wada N, Kakizawa Y. Clinical characteristics of aneurysmal subarachnoid hemorrhage in the elderly over 75; would temporal muscle be a potential prognostic factor as an indicator of sarcopenia?. Clin Neurol Neurosurg. 2019. 186: 105535
27. Kim HC, Rhim JK, Ahn JH, Park JJ, Moon JU, Hong EP. Machine learning application for rupture risk assessment in small-sized intracranial aneurysm. J Clin Med. 2019. 8: 683
28. Kobayashi Y, Kobayashi Y.editorsJapanese Stroke Databank 2015. Tokyo: Nakayama Shoten; 2015. p.
29. Kumar R, Gupta A, Arora HS, Pandian GN, Raman B. CGHF: A computational decision support system for glioma classification using hybrid radiomics-and stationary wavelet-based features. IEEE Access. 2020. 8: 79440-58
30. Lao J, Chen Y, Li ZC, Li Q, Zhang J, Liu J. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep. 2017. 7: 10353
31. Lin N, Cahill KS, Frerichs KU, Friedlander RM, Claus EB. Treatment of ruptured and unruptured cerebral aneurysms in the USA: A paradigm shift. J Neurointerv Surg. 2012. 4: 182-9
32. Molyneux A, Kerr R, Stratton I, Sandercock P, Clarke M, Shrimpton J. International subarachnoid aneurysm trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: A randomised trial. Lancet. 2002. 360: 1267-74
33. Niftrik CH, der Wouden F, Staartjes VE, Fierstra J, Stienen MN, Akeret K. Machine learning algorithm identifies patients at high risk for early complications after intracranial tumor surgery: Registry-based cohort study. Neurosurgery. 2019. 85: E756-64
34. . Report of world federation of neurological surgeons committee on a universal subarachnoid hemorrhage grading scale. J Neurosurg. 1988. 68: 985-6
35. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstem AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996. 49: 1373-9
36. Rinkel GJ, Algra A. Long-term outcomes of patients with aneurysmal subarachnoid haemorrhage. Lancet Neurol. 2011. 10: 349-6
37. Risselada R, Lingsma HF, Bauer-Mehren A, Friedrich CM, Molyneux AJ, Kerr RS. Prediction of 60 day case-fatality after aneurysmal subarachnoid haemorrhage: Results from the international subarachnoid aneurysm trial (ISAT). Eur J Epidemiol. 2010. 25: 261-6
38. Rosengart AJ, Schultheiss KE, Tolentino J, Macdonald RL. Prognostic factors for outcome in patients with aneurysmal subarachnoid hemorrhage. Stroke. 2007. 38: 2315-21
39. Rubbert C, Patil KR, Beseoglu K, Mathys C, May R, Kaschner MG. Prediction of outcome after aneurysmal subarachnoid haemorrhage using data from patient admission. Eur Radiol. 2018. 28: 4949-58
40. Shiue I, Arima H, Hankey GJ, Anderson CS. Location and size of ruptured intracranial aneurysm and serious clinical outcomes early after subarachnoid hemorrhage: A population-based study in Australasia. Cerebrovasc Dis. 2011. 31: 573-9
41. Sony Network Communications, Prediction One. Available from: https://www.predictionone.sony.biz [Last accessed on 2020 Mar 29].
42. Staartjes VE, Stumpo V, Kernbach JM, Klukowska AM, Gadjradj PS, Schröder ML. Machine learning in neurosurgery: A global survey. Acta Neurochir (Wien). 2020. p.
43. Steindl A, Leitner J, Schwarz M, Nenning KH, Asenbaum U, Mayer S. Sarcopenia in neurological patients: Standard values for temporal muscle thickness and muscle strength evaluation. J Clin Med. 2020. 9: 1272
44. Temko A, Sarkar A, Lightbody G. Detection of seizures in intracranial EEG: UPenn and mayo clinic’s seizure detection challenge. Annu Int Conf IEEE Eng Med Biol Soc. 2015. 2015: 6582-5
45. .editors. Japanese Guidelines for the Management of Stroke 2009. Tokyo: Kyowa Kikaku; 2009. p.
46. .editors. Japanese Guidelines for the Management of Stroke 2015. Tokyo: Kyowa Kikaku; 2015. p.
47. Toledo P, Rios PM, Ledezma A, Sanchis A, Alen JF, Lagares A. Predicting the outcome of patients with subarachnoid hemorrhage using machine learning techniques. IEEE Trans Inf Technol Biomed. 2009. 13: 794-801
48. San Francisco: Kaggle. Available from: https://www.kaggle.com/c/seizure-detection [Last accessed on 2020 Jun 28].
49. Witsch J, Frey HP, Patel S, Park S, Lahiri S, Schmidt JM. Prognostication of long-term outcomes after subarachnoid hemorrhage: The FRESH score. Ann Neurol. 2016. 80: 46-58
50. Zeiler FA, Lo BW, Akoth E, Silvaggio J, Kaufmann AM, Teitelbaum J. Predicting outcome in subarachnoid hemorrhage (SAH) utilizing the Full Outline of UnResponsiveness (FOUR) score. Neurocrit Care. 2017. 27: 381-91