Abstract

The principles of peer reviewed scientific publications date back two and one-half centuries to the origins of Medical Essays and Observations published by the Royal Society of Edinburgh (1731). This year (2012) is notable in that perhaps the most prestigious and best-known medical journal, the New England Journal of Medicine, crossed the second century mark. The methodologies of peer review have undeniably served medicine well and helped to usher in unimaginable advances in human health. Despite such illustrious history, the winds of change are in the air.

Keywords: Crowd sourcing, medical journals, peer review, scientific journals

Peer review is grounded primarily in the premise that acceptance by one's colleagues, and especially one's intellectual counterparts and rivals, is needed to validate the science being reported. Despite this widely accepted premise, it is worth a reminder that arguably the most important scientific publication of the 20^th century, Albert Einstein's “Annus Mirabilis,” which single handedly reordered the world of modern physics, was never subjected to independent “peer review.” Perhaps given the societal need to discredit “quack” medicine, which could possibly jeopardize the public health, a stronger argument can be made for the necessity of subjecting medical science to peer review. However, even after two centuries of success, an important question arises.

Do the past traditions of peer review medical journals continue to serve society well or are they about to be eclipsed by newer and even more powerful opportunities for establishing scientific validity?

PROBLEMS WITH PEER REVIEW

Despite its importance in our world of science, the methods of peer review have been subjected to rather little self examination. Given the highly subjective processes at play, which do not lend themselves to simple analysis, the paucity of empirically validated peer reviewed studies examining peer review itself is not surprising. However, several recent studies have highlighted major flaws in the process. For example, John Ioannidis (2005) provides convincing evidence that despite peer validation, many, if not most, major published clinical studies are ultimately disproven despite extremely rigorous review by a number of the most prestigious medical journals.[ 1 ] Recent critiques have also shown that those journals with the highest impact factor, which are widely believed to subject manuscripts to some of the most rigorous peer review prior to publication, have had an alarmingly high and growing number of papers retracted due to scientific fraud.[ 3 ] Compounding this problem, and somewhat surprisingly, retracted papers continue to be cited by subsequent publications. In 94% of these citations, no mention is made about the reference in question being withdrawn.[ 4 ] As a consequence, an impression lingers that once published in the peer reviewed literature, especially journals with high impact factors, a paper is afforded unassailable “biblical”-like authority. Pressures which stem from the desire for professional advancement, grant funding and the need to build a referral base for clinical practice all seemingly conspire to encourage such dishonest science. Unfortunately, the only practical counterbalance to safeguard the validly of scientific publishing, at least in the short term, relies on the intrinsic honesty of authors; even the most diligent of peer review processes cannot discern fraud without access to raw data, and probably also the ability to audit the collection of such data. Only the collective inability to reproduce scientific findings over time can reliably enable careless or dishonest medical science to be exposed. If this is true, what purpose do many of the rituals of the current peer review process serve? Regardless, these circumstances do serve to illustrate how difficult it can be for even the most conscientious and time consuming peer review methods to detect and root out dishonest science and flagrant “quackery.”

REVIEWER BIAS

“The beauty of science” is, by the virtue of its often illusive nature, in the eye of the beholder, and therefore inevitably subject to human prejudices. Although the existence of such bias is widely acknowledged, this assertion does not lend itself to simple empirical measurement. But even in the setting of so-called blind reviews, it would seem unlikely that any reviewer can consistently arrive at complex qualitative decisions without some implicit bias seeping into the process. Combining innate fallibility with the academic enterprise that underlies much of peer review, about which Henry Kissinger is widely quoted as saying “university politics are so vicious precisely because the stakes are so small,” it would seem the antithesis of human nature to believe that even blinded reviewers, who are typically chosen from a small pool of familiar intellectual competitors, will consistently arrive at prejudice-free judgments. In support of this contention, data indicate that research papers submitted by women and non-academics are statistically more likely to be negatively impacted by the traditional peer review process.[ 5 ] Such outcomes further impede women's advancement up the “academic ladder,” a phenomenon that has been well documented.[ 6 ]

While it has been suggested that strong supervision by the editor-in-chief can undo a tendency toward bias, one cannot help but notice that those who make such arguments have been editors-in-chief's, who having often dedicated many years of their lives to a journal, tend to be most heavily invested in the existing system of peer review.[ 7 8 ] Meanwhile, those of us who throughout our careers have reviewed, and been reviewed, in the time honored system of peer review, can testify that well-documented reviewer bias is merely the tip of the proverbial iceberg. After nearly three decades of medical academia, my own observations in this regard mirror the advice I was given during residency by two intellectual giants of 20^th century medicine, neurologist C. Miller Fisher and neurosurgeon Lars Leksell. Both men reported to me how their best ideas had been subjected to the slings and arrows of the peer review process, especially early in what would become illustrious careers, and as a consequence each advised me to ignore petty reviewer jealousies and to be content to publish important ideas within comparatively obscure medical journals. The message from these titans was that even the most important ideas can go unappreciated, or worse, be discriminated against, by a peer review process that inherently reeks of Henry Kissinger's admonition.

It is worth noting that reviewer, and thereby journal, rejection is rarely the end of the line for a determined author. Should a prestigious high impact journal reject a paper as unsuitable, for whatever reason, an author is always free to subsequently submit it to a lower status journal; and, if necessary, this process can be repeated several times until some cohort of reviewers deems a paper acceptable for publication. Once published, the history of rejection vanishes and for all eternity the article in question now bears the seal of “peer reviewed” and is therefore ascribed unique credibility. Looking back at the process, neither an author nor anyone else can know if rejection was ultimately based on the quality of the science, general interest in the topic at hand or reviewer bias. The problem with the above scenario is its gross inefficiency. The time and effort required to submit, edit, and then resubmit, only to get rejected, and then repeating the process over again, even multiple times if necessary, is often a test of endurance and will, not necessarily intrinsic scientific quality. How many great ideas never see the light of day having been defeated by the methodologies inherent to traditional peer review publishing? Such inefficiencies seem antiquated and socially unproductive in a modern world impatient for answers to our medical afflictions.

Despite the biases steeped in traditional peer reviewed publishing the process does have its pluses. Specifically in the world of paper journals, which have a finite number of printed pages, the rejection process, even if sometimes arbitrary, does provide a needed method for rationing a limited and expensive (including its environmental impact) resource. Moreover, the process of expert review can lead to useful suggestions, which if incorporated in a manuscript, can ultimately make for a better publication. However, might there be an alternative solution to these above-mentioned challenges?

MEDICAL JOURNAL BALKANIZATION

Most journals of medicine are sponsored and organized by medical societies. At the time of their formation, members of these “social networks” share common and unique intellectual interests which the journals reflect. However, over time, sub-specialization within a field increasingly disperses what were once common scientific interests and in turn the clinical practices and research interests of adjacent medical fields start to intersect with one another. These newly created niches are rarely mirrored within the social networks and subject matter of neighboring journals. Instead, parallel silos of similar knowledge grow up rigidly alongside one another with limited ability for cross communication. This unproductive structure is reinforced by medicine's worship of tradition as well as the lucrative economics of medical journals, which serve to underwrite many functions of medical societies, including their increasingly political agendas. This combination of forces has resulted in an increasingly balkanized universe with content spread across more than 6000 (and growing) individual journals. Such proliferation has made it harder for readers of medical journals, including an emerging class of avid-patients, to find the relevant content they seek. Figure 1 , illustrates the challenges of reader engagement and finding relevant content in an environment of paper journals.

Figure 1

This image of unopened mail, taken from the office of a chaired professor at a leading American university, illustrates the challenges all physicians confront in consuming the current torrent of paper medical journals

Virtually, every research topic, and thereby related scientific paper, has a natural constituency (often best expressed by key words), which if given the chance, might ordinarily cross medical specialties. Is it possible that the world of internet search engines and communication tools can enable a new peer review process that more readily reflects the realities of modern medical specialization?

THE INTERNET: A REVOLUTION IN COMMUNICATION

As noted above, peer networks have historically provided the foundation for nearly all scientific journals; social grouping of individuals with common interests (i.e. editorial boards) collectively joined efforts to attract and curate scientific papers, thereby ensuring the quality of the traditional medical journal as we know it. Meanwhile, over the past two decades the internet has emerged as one of the great revolutions in human communication. In particular, the immense breadth and depth of the internet has necessitated the development of tools that enable widely dispersed individuals to efficiently identify one another, converse, and collectively work together on projects of common interest, thereby fostering the concept of online “social networks.”

Given the parallels between medical journals and online social networks, could these new internet techniques have direct bearing on the process of peer reviewed journalism?

REIMAGINING PEER REVIEW

Despite the shortcomings of traditional scientific peer review, the most counterproductive of which involve reviewer bias, the process does clearly provide some benefits, which include:

Mechanism for rationing the limited space within paper journals

Fortunately new internet methodologies make it possible to re-imagine, expand on, and enhance peer review. First, limited space has become a non-problem on the internet. The costs associated with storing even large documents with detailed images, and even video, are becoming less and less expensive, almost to the point of becoming free. As a result, there is no reason to reject any reasonably credible manuscript from being published simply to conserve journal space. Moreover, once published, internet search tools enable relatively straightforward and expedient processes for locating papers on a topic of interest.

Second, given the capacity of the internet to publish and identify a near limitless amount of published content, reviewers of new articles have the opportunity to redirect their energies away from rationing journal space to simply improving the manuscripts they review. With paper rejection no longer in the equation, the problem of reviewer bias begins to dissipate. The job of the reviewer evolves to its intended role, to recommend changes to a paper which should (theoretically at least) improve the overall quality of the manuscript. In this new model, the reviewer works alongside the author, much like a book editor, to produce the best possible scientific paper. This symbiotic effort strives to make the author's novel concepts, information and methodologies more intelligible to readers rather than deconstruct and reject that which does not meet oftentimes arbitrary (yet stringent) criteria for publication.

However, can such a model for peer review result in scholarly quality?

Although not peer reviewed in the traditional sense of most medical journals, Wikipedia is an example of vast knowledge being documented and shared by harnessing the collective power of individuals with common interests and where only a loose editorial authority is needed to discipline the process.[ 9 ] Another example of a new and more open procedure for peer reviewed science, which has been widely embraced in theoretical physics and mathematical circles, is a journal called ArXiv. Papers are uploaded onto the www.ArXiv.org website without any prior formal peer review, but after manuscripts have been electronically published, interested researchers are invited to critique, comment and debate on them. Within ArXiv, peer review is an open, post publication “wrestling” process, that more closely mirrors the philosophical workings of science.

Third and lastly, any new method of peer review must address the need to provide a framework for assessing the quality of published papers. It is in this realm that the internet now provides an amazing array of tools, the most powerful of which stem from the collective intelligence of large numbers of individuals and termed “crowd sourcing”. In his book “The Wisdom of Crowds”, James Surowiecki, describes how large groups of individuals, when working together on a problem provide sufficient statistical power to arrive at answers that defy individual efforts.[ 10 ] For example, individual estimates of the number of jelly beans (or coins) within a large jug can vary widely, but when a large number of such guesses are averaged, the final answer tends to be very close to the true value. On the internet it is now common place for social networks like Yelp.com and Youtube.com to survey consumers about a range of topics (e.g. the quality of restaurants, books, videos, etc.). By computing an average value for such responses, myriad websites now guide users through a wide assortment of purchase decisions. Although there are clearly limitations to such recommendations, (and even well documented abuses) especially when the number of responses limits statistical power, consumers within modern society have come to routinely rely on the authority of such “crowd sourcing”.[ 11 ] In many ways, the idea of a collective intelligence harkens back to Abraham Lincoln's discerning observation that “You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time.”

Online resources such as Wikipedia also represent the collected intelligence of the many. Although no over-arching authority is in charge of ensuring the veracity of Wikipedia entries, a reasonably careful analysis published in Nature in 2005 concluded that the information in Wikipedia was on average as reliable, if not more, than the encyclopedia Britannica.[ 2 ] These examples show that “crowd sourcing” consensus can be self-correcting and can help ensure a high measure of quality information. If so, it would seem not too big an extrapolation to suggest that such principles could be useful in assessing the quality of scholarly papers, as opposed to merely relying on the judgments of two “expert” and not so dispassionate reviewers.

A NEW WAVE OF MEDICAL PUBLISHING

The present article set out to build a rationale for a new model of peer reviewed journalism that harnesses the power of the internet to accelerate the curation, dissemination, and discovery of scientific knowledge. The hypothesis being proposed is that a more efficient model is possible, one which simultaneously also enables the quality of individual published papers to be more accurately measured. By employing the scientific method, the hypotheses underlying this new type of peer review can be empirically tested. If the arguments being made here are valid, then over time, the proposed crowd sourcing methods for judging quality should be reflected in the citation index of each paper.

To specifically investigate the arguments presented herein, a new online medical journal has been launched recently. This journal, named Cureus (pronounced Curious), has been designed to incorporate the new concepts espoused in the present dialogue. Cureus uses a crowd sourcing method for establishing the quality of scientific papers which is termed Scholarly Impact Quotient or SIQ. Published articles are scored by both the experts who reviewed the paper prior to publication and a diverse cross-section of users, a number that could theoretically number in the thousands over time. The cumulative score, SIQ, is intended to be “an evolving, yet enduring reflection of a paper's true scientific impact as judged by the medical community at large. Although non-experts can rate papers, the scholarly impact calculations place special emphasis on areas of specialization and the peer reviewed publication record of scoring experts.” Readers are encouraged to visit the Cureus website (www.cureus.com) and register.

For KG4

Budd JM, Coble ZC, Anderson KM. “Retracted publications in biomedicine: Cause for concern.” Association of College and Research Libraries National Conference Proceedings 2011;390-95.

For KG5

Guynn J, Chang A. “Yelp reviews: Can you trust them? Some firms game the system.” Los Angeles Times, 4 July 2012. Web. 1 October 2012.

For KG6

Naik G. “Mistakes in scientific studies surge.” Wall Street Journal, 10 August 2011. Web. 1 October 2012.

For KG7

Pololi LH, Civian JT, Brennan RT, Dottolo AL, Krupat E. “Experiencing the Culture of Academic Medicine: Gender Matters, A National Study.” J Gen Intern Med. 2012 Aug 31. [Epub ahead of print] PMID: 22936291

For KG8

Winnik S, Raptis DA, Walker JH, Hasun M, Speer T, Clavien PA, et al. From abstract to impact in cardiovascular research: Factors predicting publication and citation. Eur Heart J. 5 June 2012 [Epub ahead of print] PMID: 22669850

References

1. Apuzzo M. Peer-review: A citadel under siege. Neurosurgery. 2008. 71: 555-6

2. Last accessed on 2012 Oct 1. Available from: http://www.Wikipedia.com .

3. Budd JM, Coble ZC, Anderson KM. Retracted publications in biomedicine: Cause for concern. ACRL. 2001. p. 390-5

4. Budd JM, Coble ZC, Anderson KM. “Retracted publications in biomedicine: Cause for concern.” Association of College and Research Libraries National Conference Proceedings. 2011. p. 390-95

5. Guynn J, Chang A. Yelp reviews: Can you trust them? Some firms game the system. Los Angeles Times. 2012. p.

6. Ioannidis JP. Why most published research findings are false?. PLoS Med. 2005. 2: e124-

7. Naik G. “Mistakes in scientific studies surge.”. Wall Street Journal. 2011. p.

8. Pololi LH, Civian JT, Brennan RT, Dottolo AL, Krupat E. “Experiencing the Culture of Academic Medicine: Gender Matters, A National Study.”. J Gen Intern Med. 2012. p.

9. Relman AS, Angell M. How good is peer review?. N Engl J Med. 1989. 321: 827-9

10. Surowiecki , James . The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business. 2005. p.

11. . From abstract to impact in cardiovascular research: Factors predicting publication and citation. Eur Heart J. 2012. p.