Abstract

Background: In regard to scientific information, are we effectively reaching the universe of physicians in the 21st century, all of whom have different backgrounds, practice environments, educational experiences, and varying degrees of research knowledge?

Methods: A comparison of the top nine neurosurgery journals based on various popular citation indices and also on the digital metric, Readers (Users)/month, was compiled from available metrics and from internet sources.

Results: Major differences in the ranking of the Readers (Users)/month metrics compared to ranking of the various citation indices were found. It is obvious that the citation indices do not measure the number of readers of a publication. Which metric should be used in judging the value of a scientific paper? The answer to that question relates to what the interest of the reader has in the scientific information. It appears that the academic scientist may have a different reason for reading a scientific publication than a physician caring for a patient.

Conclusions: There needs to be more than one type of metric that measures the value and “Impact” of a scientific paper based on how physicians learn.

Keywords: Citation indices, Comparison publication metrics, How do physicians learn? Readers (Users) month

INTRODUCTION: THE DEVELOPMENT OF SCIENTIFIC PUBLISHING

History of development of scientific journals

Wikipedia states, “The history of scientific journals dates from 1665, when the French Journal des sçavans and the English Philosophical Transactions of the Royal Society first began systematically publishing research results. Over a thousand, mostly ephemeral, journals were founded in the 18^th century, and the number has increased rapidly since then.”[ 16 ] “Prior to mid-20^th century, peer review was not always necessary, but gradually, it became essentially compulsory.”[ 16 ] Scientific articles allow researchers to keep up to date with the developments in their fields, better allowing them to direct their own research.[ 16 ] In medicine, scientific papers have been used to communicate clinical observations that often provide the basis for future scientific work; “An essential part of a scientific article is citation of earlier work.”[ 16 ]

The growth of journal publishing

From a paper written in 1990, “The number of scholarly journals in all fields (scientific and others) has risen from 70,000 to 108,590 over the past 20 years, according to the Bowker/ Ulrich’s database”…[ 10 ] “Institute for Scientific Information (ISI) database covers only the top science and social science journals – some 4,500 out of nearly 74,000 scientific titles are listed in Bowker/Ulrich’s database, a commercial listing of all periodicals; the average member of the Association of Research Libraries, in 1990, holds only about 27,000 titles, about 26% of the total available (published).”[ 10 ]

The business of scientific journals

Scientific papers are usually published by private publishing companies, which charge a fee for either a subscription to a journal or to obtain single papers. The scientific content is critically reviewed by reviewers, picked by the sponsoring organization or society. Based on those reviews, manuscripts are either accepted, revised, or rejected. The publishing houses provide the electronic systems for coordinating the Peer Review process and the mechanisms for publishing scientific papers. These publishers receive revenues for those services. In addition, the sponsoring societies or organizations also receive revenues from the publication of the manuscripts as part of the subscription prices and advertising sold.[ 5 ]

Beginning of Open (Free) Access to scientific papers for all

In the past, access to scientific articles was only possible for those who bought subscriptions to a journal or whose university, hospital, or medical center purchased the subscriptions for their associated physicians to use. Notably, others, who did not have this access, were prevented from reading these scientific papers unless they subscribed to the journals, which was costly to the reader. The more journals that a physician reads the higher the subscription costs, thus potentially limiting the number of journals a private user could read. The establishment of internet-based scientific journals allowed “Open (Free) Access” to all internet-based journals to everyone, everywhere, at no cost. With the internet, and Open (Free) Access publishing more journal information became universally available.

Open access publishing has since flourished

“Open Access” publishing has since flourished in the interim. Although it has taken many different forms, the predominant financial impact has been to shift costs from the readers, (from personal, institutional, or organizational subscriptions), to the authors for publishing their papers.[ 3 ] Open Access publishing reduced or eliminated subscription revenues and also allowed rapid online publication of scientific papers rather than a delayed print publication process.[ 3 ] Notably, about 50% of scientific manuscripts are now published “Open Access” and “are freely available within a year or two of publication.”[ 12 ] Approximately 12% of the papers are freely available immediately on publication and another 32% after a 6–12 months delay. Sixty-one percent of biomedical research papers were freely available in some form.[ 12 ]

Cost to publish in open access journals

The author costs to publish in open access journals range from 100 to 1000 of dollars.[ 3 ] The lower authors’ fees are used to cover the technical costs of publishing in the internet published journals. Alternatively, in this transitional period, some of the higher fees charged by publishing houses are still returned to support the parent organizations (publishing houses and trade medical/surgical organizations). In the push to make the scientific literature Open Access, small scientific societies feared that they could suffer damage from loss of publishing revenues. Many rely on subscription revenue from their journals to fund other activities. The big commercial publishers have the larger size and profits to absorb financial losses in some of their journals, while the societies who publish a small number of journals may suffer a significant loss in revenue to Open Access publishing.[ 5 ]

National Institutes of Health (NIH) demands open access by 2024: scientific information for all immediately and free

If the research performed comes from public funds through a government grant, the NIH believes that this information should be freely available to readers everywhere within 12 months of publication.

The NIH Public Access Policy implements Division F. Section 217 of PL 111-8 (Omnibus Appropriations Act, 2009). The law states:

“The Director of the “NIH” shall require in the current fiscal year and thereafter that all investigators funded by the NIH submit or have submitted for them to the National Library of Medicine’s PubMed Central an electronic version of their final, peer-reviewed manuscripts upon acceptance for publication, to be made publicly available no later than 12 months after the official date of publication: Provided, that the NIH shall implement the public access policy in a manner consistent with copyright law.”

How “open access” impacted the journal publishing business

The shift to internet open access publishing allowed more space to publish material than previously available in printed journals and represented competition to regular publishing houses in making more space available for communication of scientific information, at no additional cost for publication on the internet. To ease the financial burden of the loss of revenue on the publishers, a transitional plan to “Open Access” by 2024 was devised. “Plan S, due to begin in 2021, requires researchers funded by participating agencies to ensure that their papers are free to read on publication. To ease the transition, the plan allows authors to publish in a ’hybrid’ journal, with a mix of free and pay-walled content, but only if the publisher commits to shifting the journal to entirely open access by 2024.”[ 4 ] The demand for free access to scientific papers funded by government sources, thus, threatened to reduce the revenues of publishing houses and their very existence. Furthermore, large publishing houses continue to raise their charges to the major customers: universities, hospital systems, or National Governmental organizations that desire access to the large number of journals which these publishing houses control.

Countries and large universities urge immediate publication from large publishing houses while demanding less charges

“Universities fear they could end up paying more to help their scientists publish their work than they do now for bulk subscriptions from the publishers.” Large journal users such as universities or hospitals, would not only be paying bulk subscription costs to access the number of journals they buy for their readers but also the added author costs for publication of scientific papers.[ 3 ] “In some instances, NIH research grants are used to cover these publication costs.” “A consortium of Norway’s major research institutions … canceled all subscriptions with publishing giant Elsevier, joining a global push for immediate free access to scientific journal articles… … German and Swedish institutions, and the University of California cancelled its library subscriptions with Elsevier,” as reported in Science: 363:1255, 2019.[ 11 , 13 ] This issue has had some temporary solutions, but the final resolution of this conflict has yet to be achieved. Open Access publishing continues to grow.

INDEXING SYSTEMS REFLECTING CITATIONS OF PUBLISHED PAPERS

How do indexing systems work and how are they used?

Indexing systems were developed to relate the number of cited published papers over a defined time period to the total number of papers published by the journal in that same period. This figure was known as the “Citation Index.” These indices were used as a measure of the importance of a paper and of the journals in which those papers were published. However, these indices did not measure the number of readers of a paper. With the internet, the actual number of readers of a journal and the number of times an individual paper is read can be measured electronically. It appears that the “Citation Index” is no longer the sole determinant of a paper’s quality or impact. Rather, other measures are needed and already exist that also reflect interest in a publication, for example, the number of readers who read or download a paper.

The citation indices were used (a) to assess the number of times a paper was cited, but were also used and (b) in academic credentialing for promotion in academic ranks based in large part by the number and impact of scientific articles published. In addition, (c) public funding bodies often require the results of scientific research to be published in scientific journals that have high citation indices.[ 16 ]

How often are papers cited?

In the paper by Hamilton form 1990, he states, “Citations, according to the conventional wisdom, are the glue that binds a research paper to the body of knowledge in a particular field and a measure of the paper’s importance. So what fraction of the world’s vast scientific literature is cited at least once?”…[ 10 ] Statistics compiled by the Philadelphia- based ISI indicate that 55% of the papers published between 1981 and 1985 in journals indexed by the institute received no citations at all in the 5 years after they were published. The figure was derived by ISI analyst David Pendlebury, who at the request of the journal, Science, searched ISI’s extensive database of scientific citations.…[ 10 ] The conventional wisdom in the field is that 10% of the journals get 90% of the citations… “These are the journals that get read, cited, and have an impact.”[ 10 ]

Do the indexing systems reflect the interests of those who are searching for information on patient care? Or do these grading systems only reflect a subset of readers consisting of academics?

“To critics of the academic promotion system like the University of Michigan President James Duderstadt, the growing number of journals and the high number of uncited articles simply confirm their suspicion that academic culture encourages spurious publication” Duderstadt stated, “It is pretty strong evidence of how fragmented scientific work has become, and … the kinds of pressures which drive people to stress the number of publications rather than quality of publications.” Duderstadt said that most of that pressure is rooted in the struggle for grants and promotions. “The obvious interpretation is that the “Publish or Perish” syndrome is still operating in force,” said David Helfand, chairman of the Astronomy Department at Columbia University.[ 10 ]

Who reads a scientific paper and why?

The research scientist looks for papers that are related to his/ her research interests and will use those papers in his/her scientific reporting. The clinician-scientist looks at articles for information that will affect his/her practice but that he or she may not necessarily use in publishing a paper. Does that distinction make any article less important or have less “impact?” What, therefore, is the proper metric to be used to evaluate a manuscript’s value to the reader? And does the grading system bias the selection against articles considered less important to bolster the journal’s perceived citation grade? Should scholarly productivity for clinicians be measured using similar strategies as are used in basic science research or the life sciences? In other disciplines, “research for research’s sake” is acceptable, but in medicine, research should always have the end goal of patient care and translation to the bedside.

How do physicians learn?

This paper compares the metrics of the different citation indices with the data of Readers(Users)/Month as another measure of readership. Who reads medical journals and Why? How do physicians learn? The purpose of this paper is to provide data and metrics on some of the most common neurosurgical journals that are published, so the reader can decide the relative values of each journal for his or her needs and to provide a focus on the system of evaluation of scientific publications.

METHODS

List of neurosurgical journals and request for metrics

A list of the neurosurgical journals was obtained from Google Scholar and other citation databases. The Google ranking system for 20 neurosurgical journals was used as the basis for the listing of journals. The website of each journal was examined to provide the metrics sought in this paper. Those metrics included the publishing organization, the journal title, the number of Readers(Users)/Month, the Google Citation index factor mean and median,[ 9 ] and the SLR Ranking score and rank,[ 14 ] “Impact Factor” scores obtained from Clarivate Analytics,[ 7 ] owner of the original Thomson Reuters publication of Impact Factors. Other ranking systems use the words impact factor but is an “unofficial” analyses of data, according to Surgical Neurology International (SNI’s) publisher. The rank orders were determined for each grading system from the data provided. Other information recorded in Table 1 included: Open Access status: Free and Costs (whether the papers would be free to the readers or required some payment); Articles of Interest: the type of papers the journal would like to accept; Online Circulation: the distribution of readers worldwide by percentage; Subscription Costs; Author Costs; Time from date of acceptance to first Online or Print publication; Web address for submitting a paper; and Publisher.

Entries for cited journals verified by managing editors or editors

Table 1 was constructed utilizing the metrics obtained from Google Scholar and other citation databases. It was then sent to the Editor or Managing Editor of each journal for their verification and additions. Each entry of the cited journal was verified by the Managing Editors or Editors with each revision. After several revisions and updates, the final table and text were re-sent to all the journal Editors or Managing Editors for their final approval before publication. Data that were not available from a journal are listed as a blank entry in the table. The sources of the Readers(Users)/Month data are cited in the footnotes. Data from the other journals included in the 10–20 lists in the Google Scholar ranking were not reported at this time, but will be reported in a later publication for completeness. Most of these journals in the 10–20 list appealed to local or regional markets, and in most cases had lower rankings than those journals reported in Table 1 .

Table 1A:

Neurosurgery journals metrics comparisons as of April 19, 2020.[ 10]

Table 1B:

Neurosurgery journals metrics comparisons as of April 19, 2020.[ 10]

RESULTS AND DISCUSSION

Table 1 contains all the metrics used for this paper for the journals cited. The list in no way makes any judgments about the quality of the various journals or their value to their audiences.

Readers (Users) per month is not a standardized metric

It is immediately evident that the ranking of the journals varies depending on the metric used to evaluate them. We asked for the number of Readers (Users) each month from the listed journals’ editors. Various answers were given including: the full text downloads, pdf files/month, downloads per year/12, total Readers/month, html versions, or new Readers (Users)/month. Others used different metrics. Each journal reported the Users/Readers/month metric in their own way as is indicated in their footnotes to Table 1 . Some numbers may represent inflated numbers for Readers(Users)/Month by counting abstracts, downloads, and html hits as separate hits. Using downloads only may underestimate the number of readers who do not download the paper but read it on line. Thus, Readers(Users)/Month is not a standardized metric which all follow. The numbers in Column D of Table 1 should be interpreted with that understanding in mind.

Three major indexing systems: Google Scholar, Impact Factor, and SJR had different results for specific journals.

Indexing systems have been used as a measure of how many citations a journal received for its publication. However, with the introduction of the internet, actual data can now be determined regarding the number of Readers(Users)/Month, Total readers of specific papers, downloads, and locations of readers, as used in this table.

“How do we rank the importance of a journal or the journal’s value to its readers?”

Before the Internet

Given these different types of information, the question then becomes, “How do we rank the importance of a journal’s value to its readers?” The answer depends on what the reader is looking for in the journals they read. Before the internet, the citation indices, used by the academic community, were based on the number of citations a journal received. There was no way to know how many people read the journal or a paper from these Indices. Second, it was impossible to determine from the indexing systems the readers who read the paper but did not use it as a citation in a paper they may write.

After the internet

After the internet, there was an opportunity to learn exactly how many people accessed a specific journal or its articles. Furthermore, there were more journals published from which needed information could be obtained by readers. For example, those looking to publish a paper would likely be interested in publishing it in a journal with a higher citation index, as well as with a faster publication time. So, publication time becomes a metric in which readers are interested. Hence, which citation index should be used? Is it more important to know that an article is read by large numbers of people or by its citation index factor which does not measure the number of readers? If an article is read by thousands of readers even though it is not cited by another research publication, does it have less value as a publication? How do we measure that value? We do not know.

The citation indices/metrics do not truly reflect the actual number who read the paper

Table 1 shows that the citation indices metrics do not truly reflect the actual number who read a paper or use the information in the paper. Some readers are not interested in publishing journal articles, but rather, in finding useful information for (i.e. for patient care); indeed, they may not find the “cited articles” of sufficient practical interest. The published randomized controlled trials (RCT) may not be relevant to their practice. Others may live and practice in a part of the world where practical information from others’ experience is valuable, but where they need to have “Open (Free) Access” to read it. A reader may see a video of a surgical procedure or read a case report or a case series for an approach to a complex problem, but not download the paper, video, or cite the work in any publication. Yet, the information may have changed his/her practice and that of others with whom the information was shared. Is that practice the “Real Impact” of the paper? Is the “Real Impact” any less important than the number of citations a peer reviewed article receives? If we cannot measure the number of people who used the information in the paper, does that invalidate the “Real Impact” of a scientific paper?

New metrics being considered

Academic medical centers are also re-evaluating the way, the influence of scholarly work is measured. Several institutions across the country have recognized that in the digital age, the transmission of scientific knowledge is not fully appreciated when only journal citations are considered. These academic centers have started to integrate alternative metrics (altmetrics) into their assessment of the impact of scientific manuscripts, and even in their considerations for tenure. Previous authors have proposed guidelines and recommendations for the inclusion of altmetrics into the assessment of academic productivity that incorporate social media views, downloads, and followers into evaluating an academician’s overall scholarship.[ 6 ] China is also evaluating metrics other than citation indices for its journals.[ 2 ]

In the transmission of information, what do our metrics mean?

Hence, what do our metrics mean? The deeper meaning of Table 1 is, “What are we evaluating and Why?” What do the numbers mean, and how should we use them? Members of academia may look at the metrics from one perspective, but probably a much larger number who read the articles or journals for information may have very different perspective. The relative differences in the numbers listed in Table 1 are not significant, practically. The real question is what do the various metrics in Table 1 mean?

Are case reports anecdotes or are they scientific evaluations of unusual observations in clinical practice?

As an example, are case reports “anecdotes,” a word commonly used in scientific discussions about these scientific reports? Merriam-Webster defines the word anecdote as “a usually short narrative of an interesting, amusing, or biographical incident,”[ 1 ] a term that has been used by the academic community to degrade the significance of the observations as not being more detailed and validated scientifically and perhaps questionable. Case reports or case series are scientific records of experience and observations that form the foundation of advances in natural sciences and medicine and report innovative observations, which often do not come from RCT. Yet, such observations provide the basis for RCT and more detailed investigations.[ 8 ] There is a tendency to disregard the case reports or case series as meaningless anecdotes and among the lowest rated medical evidence as seen on the scale of importance by Evidence Based Medicine charts. Yet, case reports are often considered valuable to practicing physicians as they contain information not found elsewhere on managing their patients. The scientific community relegates these scientific observations to insignificance. As we march down the road to bureaucratic medicine and standardization, are we demeaning the value of creativity and innovation? RCT do not incentivize creativity, but are a more regimented reporting of planned observations. What information is important to physicians?

In the coronavirus epidemic is not there a desperate need for information from case reports or small case series to propel larger studies of treatment choices? As the threat of death looms for patients, are not the case observations of great significance? Yet, the scientific community wants to wait for a vaccine to be developed or for a RCT to validate observations. Although this argument is ideal, it is not reasonable to the clinician fighting to save the life of a patient today when the ideal information is not available. People do not have years to wait for answers to near death problems as in RCTs. Is that the decision you would make for your family or patient? How do physicians learn?

In an unpublished survey of its readers, SNI found that physicians regarded, among its most valued sources of information: “Talking with Colleagues,” which was as important as reading abstracts, and which ranked higher than attendance at all types of meetings, podcasts, CME courses, and reading complete journals. How do we evaluate the metric, “Talking with Colleagues?” How do physicians learn?

How do physicians learn?

The fundamental question is, “How do physicians learn?” There is little information in the literature on this question. Are we effectively reaching the universe of physicians in the 21^st century, all of whom have different backgrounds, practice environments, educational experiences, and varying degrees of research knowledge? Are our grading systems for judging scientific papers a valid way to measure the “impact” of these papers to most physicians? Are multiple metrics in order?

Fundamentally, are papers in scientific journals are being written for the benefit of the doctor authors and institutions or for the benefit of the patients or both? Our metrics of evaluation do not reflect such a distinction is being made.

Disclaimer

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Journal or its management.

Declaration of patient consent

Patient’s consent not required as patients identity is not disclosed or compromised.

Financial support and sponsership

This paper was supported by a donation to Surgical Neurology International from the James I. and Carolyn R. Educational Foundation.

Conflicts of interest

There are no conflicts of interest.

Acknowledgements

The Authors wish to thank the following people for their advice and comments on improving and adding data to this manuscript: Miguel A. Faria, M.D., is Associate Editor in Chief in Socioeconomics, Politics, Medicine, and World Affairs of Surgical Neurology International (SNI); Edward C. Benzel, MD. Editor in Chief, World Neurosurgery; Mike Cheley, Principal, Information Architect, Graphtek CMS, Inc. Palm Desert, CA. mcheley@graphtek.com; Jo Ann M Eliason, MA, ELS(D) Communications Manager JNS Publishing Group; Brandon J. Fiedor. Director of Publications NEUROSURGERY® Publications; Dennis Malkasian, PhD, MD, Neurosurgeon, Irvine California; Tiit Mathiesen, MD. Editor In Chief, Acta Neurochirurgia.

References

1. “Anecdote.” Merriam-Webster Dictionary. Available from: https://www.merriam-webster.com/dictionary/anecdote [Last accessed on 2020 Mar 25].

2. Brainerd J. China sours on publication metrics. Science. 2020. 367: 960

3. Brainerd J. Publishers Roll Out Alternative Routes to Open Access. Available from: https://www.sciencemag.org/news/2020/03/publishers-roll-out-alternative-routes-open-access [Last accessed on 2020 Mar 05].

4. Brainerd J. Scientific societies worry about threat from plan S. Science. 2019. 363: 332-3

5. Brainerd J. New deals could help scientific societies survive open access. Science. 2019. 365: 1229

6. Cabrera D, Vartabedian BS, Spinner RJ, Jordan BL, Aase LA, Timimi FK. More than likes and tweets: Creating social media portfolios for academic promotion and tenure. J Grad Med Educ. 2017. 9: 421-5

7. Clarivite Analytics: In Cites Journal Citation Reports (Takes the Place to Thomson-Reuters Impact Factors) Journal Data Filtered By: Selected JCR Year: 2019 Selected Editions: SCIE, SSCI Selected Category Scheme: WoS. See Clarivite Analytics for JCR: May Require Sign In.

8. Coccia CT, Ausman JI. Is a case report an anecdote? In defense of personal observations in medicine. Surg Neurol. 1987. 28: 111-3

9. Google Scholar Citation Indices Mean and Median: Google Scholar Citation Indices. Available from: https://www.scholar.google.com/citations?view_op=top_venues&hl=en&vq=med_neurosurgery [Last accessed on 2020 Mar 22].

10. Hamilton DP. Publishing by--and for?--the Numbers. Science. 1990. 250: 1331-2

11. Hiltzik M. In Act of Brinkmanship, a Big Publisher Cuts Off Access to its Academic Journals. Available from: https://www.latimes.com/business/hiltzik/la-fi-uc-elsevier-20190711-story.html [Last accessed on 2020 Mar 05].

12. Kaiser J. Half of all papers now free in some form, study claims. Science. 2013. 341: 830

13. Science contributors. Norway cancels elsevier. Science. 2019. 363: 1255

14. SJR Data. Available from: https://www.scimagojr.com/journalrank.php?category2728&area=2700&type=j&page-5&total_size=360 [Last accessed on 2020 Mar 24].

15. Thomson Reuters Impact Factors. Available from: https://www.researchgate.net/publication/322601291_Journal_Impact_Factor_2017-_Thomson_Reuters [Last accessed on 2020 Mar 05].

16. Wikipedia Contributors; Scientific Journal. Available from: https://en.wikipedia.org/w/index.php?title=Scientific_journal&oldid=945583311 [Last accessed on 2020 Mar 05].