This visualization leaves off a number of nuances to the process. It should be understood that there are a number of other precursor and other events in the process.

Another over-simplification involves the four types of feedback indicated by the blue nodes in the middle. Virtually all peer review processes that this author has encountered involve the uses of forms and rubrics to elicit the feedback. These tools address issues such as the following:

• What is the paper about? Does it offer fresh research and insights? Is the writing original?
• What is the quality of the writing? Are the ideas clearly represented?
• Is the literature review fairly comprehensive (or at least sufficient)? Is the secondary research accurately summarized? Are the research sources accurately cited?
• What research method (or methods) was used? Was it relevant to the hypothesis? Was it relevant to the research question(s)?
• Was the research conducted according to professional standards in the field?
• Were basic research ethics adhered to in the research?
• Were the research findings accurately represented? Were the data visualizations accurate? The data tables? Were the statistical analyses accurately done?
• What is the quality of the analysis from the research? Are research conclusions justified? Are suggestions logical? Practical? Beneficial to the field and practitioners?
• What is the quality of the ideas for future research?
• Are the various parts of the paper / chapter included? Are the relevant issues comprehensively addressed? Are there any gaps?

From the questions, peer reviewers offer an overall assessment.

Overall Assessment
1. Accept as-is
2. Accept with revisions
3. Re-do with new peer review sequence
4. Reject outright

Stellar Work: A stellar piece of writing may have the following characteristics:

• The work acknowledges the relevant research that came before (whether the new work aligns with that or not) and selectively offers the most relevant secondary sourced information
• It follows the respected methods for research in the particular domain
• It does not simply accept the research results but involves efforts to pressure-test the results
• It moves a field forward and maybe even elevates the field (with new learning, insights, and / or practices)
• It follows the ethical principles in the field
• It does not contravene laws
• It is original
• It follows the conventions for the particular form of writing

A stellar work should be accepted right away, given how competitive it is to attract talented authors and powerful works. It is rare that any work is completely correct, and it would generally be a good idea to offer suggestions where possible—but authors should be notified if their works were exemplary but with a few areas for improvement.

The “Revise and Resubmit” (for Publication) or (for Re-consideration) Options: A wide range of possible challenges with a work may result in the revise option. The more encouraging response is to have a conditional acceptance of the work contingent on a few changes. The less encouraging response requires a full overhaul and a need to go through the entire re-submittal and double-blind revision process (with the original peer revisers or a different set, depending on policies and depending on preferences).

Some common challenges follow:
• The research itself is solid, but more elaboration of the findings would benefit the writing.
• The writing retains the residual structures from doctoral dissertations. Or the format of the work does not fit the publication. (Some authoring teams will submit a work in the locked-in .pdf format based on the publication they wanted to appear in but were rejected from.)
• The writing does not read smoothly. It sounds unpracticed and stiff. (Or in the case of multiple authors, the writing may be further integrated and smoothed.)
• The writing consists of a lot of self-plagiarizing from a prior published work.
• The author(s) does not consider a range of possible implications in the work.
• The research citations, both in-text and in the References list, are incorrectly done.
• The work follows the guidelines for academic “fair use” standards but do not follow the higher standard for commercial publication. (This could mean copyright infringement challenges.)
• There are sections of text missing or sections which would benefit from elaboration.
• Some of the data visualizations need elaboration.

If a work is resubmitted, the peer reviewers and editors often have due diligence that they need to follow in order to make sure that the necessary changes requested were actually made. This process may be helped with the reportage by the authors of the changes made and highlights of the changes in the new manuscript. (This is especially an issue for works submitted just prior to deadline, which are being considered under time pressures.)

The “Reject Outright” Option: The “reject outright” option may seem harsh on the face of it. This begs the question of when a work is sufficiently concerning that reviewers evoke the “deal breaker” option, which closes off the potential of redeeming the work. Instant deal breakers include the following:

• Any form of deception or misinformation
• Any plagiarism, a lack of originality (many publishers run manuscripts against a known database of published works; stylometry analyses may be run to identify ghostwriting)
• A foundational misunderstanding of the topic
• Poor topic focus which indicates a misunderstanding of the publication and readership
• An unredeemable fundamental problem with the research methodology or execution
• A lack of understanding of the full implications of the research
• Ethical challenges, such as the (mis)treatment of human research subjects, or anything that would not pass muster with research compliance, and 
• Alternate agendas such as promoting certain ideologies or revenge-seeking (such as “punishing” individuals in a particular university department by writing a journal about difficult learning experiences and passing that off as a form of qualitative research)

Of course, the authors themselves may retract a work at any point in the consideration process (even though that is problematic for publishers, who’ve now invested resources into vetting the work). Authors may also choose not to revise once they’re received feedback. There are various junctures in the process when withdrawals are more common. There may be other challenges here with the process, such as when authors offer multiple manuscript submissions to various publishers (which is considered unethical).

A Histogram of Editorial Responses for Educational IT Publications: To provide a sense of the varying levels of categories of responses in academic publications in educational IT, a simple “histogram” of editorial responses was created, albeit in an area chart. This shows that only about 5% of responses are an outright acceptance. A majority, some 60%, are for the authors or authoring teams to revise and resubmit, with the assumption that if their revisions met standards that the work would be accepted. About a fifth are asked to revise and resubmit with no guaranteed acceptance but only the promise of reconsideration. For this, the editors are indicating that they are willing to have the authors/author teams move on to another publication. About 15% have their works rejected outright, some even before the works are sent on to the peer reviewers. It may be that there is not good fit in the work to the publication or major issues of quality. What this visual means for the acceptance rate is a little higher than 65% based on the author’s decades in the educational publications field in a number of roles. There are assumptions behind the data as well, such as the fact that many works have not made it to the point of being submitted to publication. (For this, think about the many who have ideas for projects they want to pursue but who never actually get started or never actually get very far.) Depending on the type of publication, these editorial response values will vary greatly.



Private-Channel Comments (Reviewers <-> Editors): Reviewers are also asked to offer private-channel comments to the editor(s) or the editorial team. The private channel is used to indicate concerns that may be unseemly to share with the authors, such as comments about the (suspicious) provenance of the work or backstories. (In some fields, some authors are somewhat recognizable because of high levels of specialization in particular domains or related prior published works.)

The Tyranny (and Benefits) of High Numbers of Reviewers to One Work: Some publications require a proposed draft work to be vetted by a dozen or two reviewers. The more reviewers there are, often, the broader the range of suggestions for feedback and the more contradictions there are among the reviewers themselves. Sometimes, the “vote” on the work may be bimodal, with a lot of yay’s and a lot of nay’s. For new authors, the large amount of feedback may be distressing; in such cases, the editor(s) or editorial board has to stand in to moderate the feedback without diluting the insights. On the up side, having a large number of reviewers may result in a broader range of types of feedback. Each reviewer is a product of his / her own time (even as he or she is continuously learning), and in that sense, it helps to have a broad spectrum of reviewers with differing backgrounds and insights.

---

As a peer reviewer for some years now, I’ve created a comfortable approach. I find it helpful to do a thorough line-by-line read-through with easy-going note-taking. I try to get a feel for the work and also who the researcher (or research team) is. In the first read-through, I strive to focus on the originality of the work, the research methodology, and the implications and relevance of the research findings. It is helpful to re-read the work then for more in-depth focus on the theoretical underpinnings, the research methodology (again), the evidence and data, the logic, the statistical analytics, the data visualizations, the research sourcing, the research citations, and any potential lapses in the work. It helps to read for what is not there: gaps in information or holes in logic or unaddressed issues.

Then, the write-up for the author(s) should be thorough, with clear citations to the page, paragraph, and line. The commentary should be constructive, to enable the authors to improve on the work. The standards referred to should be clearly defined and applicable to the field. The tone should be unfailingly respectful. There should also be some flexible understanding and acceptance of a wider range of conducting research and understanding the world since the point is not to align with the peer reviewer but to surface new understandings. (The author is understood to be the potential expert on the select topic.)