A Search for "Peer Review" in the Big Data Text Corpuses of the Google Books NGram Viewer 

Peer review has always been a contentious practice. After all, it is a gateway through which researchers have to pass to reach a wider audience and to have their work read and potentially validated. Who a “peer” is (in other words, “who decides”) is also contentious given various backgrounds and subjectivities. In a post-modern age, who decides elusive “truth” may depend in part on who has access to the means of publication and access to a wider audience. What is the inter-rater reliability among peer reviewers, each with their own idiosyncrasies and subjectivities? Eisenhart (2002) suggests that the “paradox of peer review” is that it both weakens academic standards and disqualifies innovative and diverse projects simultaneously.  

For the broader public, peer review is seen as a critical hedge against researcher mistakes and even fraud. Major controversies have exploded around issues of unsupported assertions in bench research, with accusations of fraud and malfeasance (and tens of millions of dollars invested in research based on untrue premises). Investigations to such research claims have resulted in retraction of publications (by prestigious journals), ruined professional reputations, and critiques of the limits to peer review. A more recent challenge involves the submittal of machine-generated texts (such as the SCIgen debacle) that have been accepted into publication. Preventing computer-generated “papers” from appearing in a journal is partially an indicator of how far machines have come in passing the Turing test (pretending to be human intelligence-wise). In an ironic sense, there is also a reverse-Turing challenge (in which authors have to prove their humanity) when submitting a work to a journal or book. Indeed, there are researchers who test quality controls of various journals by making up characters and submitting spoof papers (Bohannon, 2013).

The argument has gone, “Had the peer review process worked, this publication would never have been published.” In retrospect after research has been spectacularly shown to be incorrect or fraudulent, other researchers often come on board to assess how such a situation may have been headed off—through research replication, re-analyses of data sets, through statistical means, through assessment of the respective institutions (labs, universities, funding organizations, and others), and meta-analyses (such as how a new work lines up with prior works and prior knowledge). The bolder the contention and the larger its implications for humanity and the world, the heavier the burden of proof. 

An Environmental Scan of Academic Research Publications: Professional publications originate in many ways. Some are launched in concert with academic conferences. Others are state sponsored. Some are created to highlight areas of corporate interest in other to promote research and advancement in the field. Publication prestige is a feature of its track record of publications, its editors, its authors, its funding, and its impact on the field, among other factors. Its acceptance rates may be seen as one indicator of its prestige (many try to appear in the publication but only a few are accepted). Achievement in academia is often rigidly hierarchical and reputation-based. Researchers build their reputations based on their discoveries and their productivity. The name order of authors on a shared work are critical, with lead authors building huge funding opportunities and lifelong legacies. (The expectations for the inclusion of names on particular research papers differ based on the particular domain. In some, those who offered peripheral support like use of their lab equipment would not be included in the bylines but only in the acknowledgements. In other domains, even small help requires inclusion as lower-level authors. Some principal investigators expect to be included as a matter of course whether they contributed to the research or writing. Likewise, some professors expect the same inclusion to their graduate students’ projects, whether they contributed directly or not.) 

Academic publications appear and disappear as needs are identified. Once a topic is no longer sufficiently in vogue to attract research dollars, researcher interest, and reader interest, the related journals quietly sunset. While journals may drive interest, they are also driven by interest—in a mixed interdependency. Some may argue that in the Web 2.0 era with its wide proliferation of open-source and open-access publications that the hierarchical and prestige-based publishing system has fundamentally changed. So far, such online publications are untried; most are disconnected from the more traditional publishing venues with strict editorial oversight. Without the ties to expertise and credibility, most open-source publications are publishers of last resort, little more than vanity spaces and a type of informal gray literature. 

The Pressures of Speed and Time: Peer reviewers (and editors and publishers) work on tight deadlines because first-to-press establishes ownership of ideas. On one end of the continuum would be the prestige from publication which may mean resources for further research; at the other end of the continuum may well be patents (expensive to attain) and R&D (research and development) implications and the promise of money in addition to the glory. (Publications themselves are very hard to monetize, even for works published in commercial venues. A common average royalty amount for the lifespan of a book is $1,000.) 

A Side Note about the Origins of This E-Book: The original impetus for creating this book arose with the need to learn Scalar for a few projects on campus. I needed to choose a topic about which I could create a quick book, which meant I had to know the topic well and have some potential tacit insights. I needed an original topic not already in the public collection (based on a quick perusal), and the topic had to be something of interest to me. There had to be a broader public which might be interested in the topic. There had to be opportunities to include multimedia to use some of the power of the platform. The topic had to be addressable within the implied word limits of the tool. The topic also could not conflict with any other work that I’d already published. Finally, since this is linked to my university email account, I decided that the topic should align with my work at Kansas State University. 

I went through a half-dozen possible topics before I chose to focus on writing about academic peer reviewing, which was a topic that met the prior requirements. Then, I was very fortunate to have colleagues and friends willing to collaborate on parts of this work.