Q: Would you please begin by explaining how you work with academic peer reviewers at EDUCAUSE? 

A:  When submissions arrive, I evaluate them for suitability and decline inappropriate papers. Those that clearly will not pass peer review but are relevant to our audience have the option for conversion to a short article instead. Those that have obvious gaps have the option for revision before review. Those revisions and papers accepted for review are posted online in a private project-management site as PDFs with identifying information removed. A request goes out to the entire review committee for volunteers to sign up to evaluate the submission. Reviewers can sign up online or by e-mailing me. Eight to 12 reviewers are sought for each paper. Reviews (using an EDUCAUSE review form) come directly to me by e-mail to maintain the double-blind review process and avoid having completed reviews influence those still in process.

Q:  How do you recruit and retain the peer reviewers? What do you look for in these reviewers?

Q:  The EDUCAUSE Reviewers committee handles peer reviews of academic submissions to the flagship magazine EDUCAUSE Review. The committee ranges from 60-90 reviewers, with members pulled from annual volunteers to the committee, individuals who contact me or EDUCAUSE colleagues to express interest, and by invitation (often to authors who have published with us). Members serve a one-year term, with Senior Reviewers serving a three-year term. I invite reviewers to become Senior Reviewers based on their performance as peer reviewers, service to the wider community, and background. I look for thoughtful, insightful evaluations with useful suggestions for the authors in revising their paper for publication (even if rejected). Reliability in delivering reviews is important, as is participation (a minimum of one to two reviews per year). I also try to balance the committee between large and small colleges, IT professionals and others (faculty, instructional designers, librarians, administrators), career interests, and qualifications to review for a practitioner audience.

Q:  In your own judgment, how often do reviewers get it right? How often do they get a work wrong? What are some influential factors in their judgment and decision-making that affect their final evaluations? Please elaborate.

A:  Per submission, reviewers are on target about 60 percent of the time, with another 30 percent taking too narrow a view and the remaining 10 percent taking too shallow a view. A major reason for having so many reviewers per paper is to get a wider range of evaluations and lessen the impact of those providing a limited view so that the conclusion accurately judges the submission’s content and quality. For example, some reviewers give too much weight to issues that can be addressed by editing and revision, which results in an unfairly low ranking. Best are the reviews that rely on the reviewers’ estimation of the work’s quality (the research validity is more important than presentation here), appeal to the audience, and judgment of potential to revise into a publishable article. The best reviewers put the audience’s needs ahead of their personal judgment on the topic, presentation, and type of research to focus on general interest and usefulness, contribution to the field, and appropriateness of research for the topic (qualitative vs. quantitative, literature review vs. case study, etc.).

Q:  What types of peer review feedback do you find most useful for authors? Why?

A:  Suggestions for revision that point out weaknesses and gaps are most useful for authors. This guidance helps the authors reorganize presentation of the material, add or delete graphics and multimedia, flesh out skimpy literature reviews, highlight important points more clearly, and focus on the information most important to this audience. Requests for missing data, more graphics to illustrate the conclusions, and significance of the results help not just the author but also readers, ultimately.

Q:  How do you mediate between the reviewers and the authors, particularly when there are very different points-of-view? How much of a say do you have in the editorial context in terms of whether a work makes it to publication? Do you already have a sense of whether a proposed draft will "make" or not? Please explain.

A:  Once the reviewers have delivered their evaluations, I pull together a letter to the authors combining the reviewers’ comments and suggestions, with an alert to the authors if the reviewers contradict each other. In those cases, the authors must choose the advice that best fits their concept of their work; some suggestions are not feasible, and others would take the authors beyond the scope of the paper. My editorial comments and suggestions are embedded and clearly noted within the letter as editorial. While composing the letter, I leave out comments that do not serve the magazine’s goals and audience. I also organize the comments and suggestions based on importance according to my judgment. There is no contact between the reviewers and the authors; all communication is between the authors and me. Again, a larger number of reviewers provides more useful suggestions for the authors.

For borderline decisions, I make the final judgment on whether to publish the paper. For interesting papers the reviewers have declined, I invite revision to a short article (not peer reviewed). A few important papers show promise but aren’t accepted; in these cases I invite the authors to revise and resubmit, then send the revision to the same group of reviewers. In the most recent case, the revision was enthusiastically accepted by the same reviewers. Few authors are willing to invest the additional effort, however, and I rarely invite them to.

After reading a submission, I have a clear idea of whether it will pass review. As noted, some I decline outright. For others I warn the authors of the gaps that will harm the evaluations and offer them the opportunity to revise to address the weaknesses before going through peer review. If the author prefers to go through review without revising, I will send it to the committee for evaluation. Only those declined but showing promise receive an invitation to revise and resubmit.

Q:  Often, you have a back story about various works that the reviewers do not have. Is your decision-making influenced by the back stories? If so, how? Could you offer a few examples?  

The back story does not influence my decision on publication unless the reviewers came to a borderline conclusion. In those cases, it might sway my judgment for or against acceptance. The back story does influence my editorial suggestions and comments included in the letter sent to the author. It also influences whether I invite authors of rejected papers to revise and resubmit.

In a very few cases, knowledge of the author through prior work and/or communications have worked against acceptance. The author’s inability to meet the reviewers’ requests or lack of valid research support argues for rejection of the paper in my mind.

In several cases, the reviewers judged a submission on grammatical and language problems and not just on the ideas, producing a borderline ranking. I chose to accept those borderline papers with clear communication to the authors that they had to go through heavy editing/rewriting for publication. Because this lessened their effort on presentation, allowing them to focus on content, they were happy to agree. The letter included notes offering editorial help for some of the reviewers’ suggestions that requested reorganization, trimming, etc.

In some cases with a back story, I have worked with the authors before and am aware of their writing abilities, openness to rewriting, and prior work. The letter these authors receive includes more editorial comments to guide their revision before publication, focusing most strongly on the reviewer suggestions I consider vital and adding requests to reference or summarize prior work unknown to the reviewers. In a recent example, double-blind peer review necessitated removing the hyperlink to a major website of ongoing work, weakening the paper. I had notified the authors it was likely to fail review, and they chose to proceed. In sending the letter of reviewers’ comments, which pointed out the gaping holes in the research, I added requests to include research I knew they had conducted. The authors revised the paper with more of their research results, and it passed a second review.

Q:  EDUCAUSE Review Online (ERO) is an electronic publication with rich multimedia aspects. Have you seen any struggles for authors in shifting from a print publication to an electronic one? What are some common challenges?

A:  Authors struggle with creation of appropriate multimedia, especially audio and video. They still think in terms of linear print presentation, with graphics explained in detail in text rather than in an interactive fashion, for example. Too many submissions are all text or text with a few simple black-and-white graphs. Authors also tend to include too many references supporting concepts not needing a formal source, when most readers don’t look at online references anyway. They can hyperlink to many sources, such as websites for their university, a product or company mentioned, and even blog entries, reducing the number of notes.

Most authors need guidance on types of media to include, such as screen captures, video demonstrations, audio quotes from end users (although text is fine), audio/video interviews with key participants, etc. If they can’t find support from colleagues, most can’t produce the media on their own – they have neither the tools nor the experience, and time can hinder appropriate media development.

The editors usually convert long paragraph lists to bullets, which adds emphasis to key points and often trims text at the same time. Editors also suggest subheadings to break up expanses of text. Although some readers will commit to a long article online, most won’t. Making an article more accessible to online skimmers helps it in many ways, but authors don’t write with that in mind. They still think of text-heavy academic publication.


Nancy Hays is editor and manager, Publishing, for EDUCAUSE, a nonprofit association and the foremost community of IT leaders and professionals committed to advancing higher education. She joined EDUCAUSE in 2000 to manage the editorial and peer-review process for EDUCAUSE Quarterly, which moved entirely online in 2009 and merged with EDUCAUSE Review and the EDUCAUSE Multimedia programs in 2012 as the association’s flagship publication. As editor for EDUCAUSE, Hays works with the editorial, content, and executive teams to determine important trends in the field of higher education information technology, solicit appropriate authors to write on different topics, and develop ideas for publication online. She also runs the peer-review committee and process and oversees the editorial and production teams for EDUCAUSE publications, from conference programs to books.

Prior to joining EDUCAUSE, Hays was group managing editor for the IEEE Computer Society, where she started as an assistant editor in 1985. As group managing editor she supervised editorial and production teams and worked with the editors-in-chief and editorial boards for IEEE Computer Graphics and Applications, IEEE Multimedia, IEEE Micro, IEEE Design and Test, and the Annals of the History of Computing. From 1980 to 1985, she held editorial jobs in technology and medical publishing.

Hays earned a master’s degree in English literature from UCLA in 1980 and three bachelor’s degrees from Oregon State University in 1977 (English, economics, and liberal studies).

Comment on this page