Who owns your review?
I get a lot of requests from journal editors to review manuscripts. As an editor at BMC Bioinformatics I frequently ask others to review papers. Sometimes I agree to review papers, especially when I have time for it and when the manuscript is in an area where I have some expertise. Sometimes the paper I end up reviewing is simply high quality science, other times the research is good but the writeup could use some work, and unfortunately a few papers that have crossed my desk have bordered on the unintelligible. No matter what the quality of the writing, when I agree to review a paper, I usually spend many hours reading the paper, thinking about the work, checking the claims, verifying that any associated software works, checking data, and eventually writing up an evaluation of the work. That’s hard work. Writing a high quality review is hard work.
Somewhat paradoxically, the reviewers who spend countless hours to write high quality reviews go largely unrewarded. They are unpaid volunteers, and the cultural practice of anonymous peer review disconnects the reviewer from receiving credit from the rest of the scientific community. The only person who ever has the opportunity to associate the reviewer’s name to the review is the journal editor and many journal editors, especially those at high profile journals, are professionals who do not participate directly in academic research (or in faculty hiring committees). One could maybe argue that high quality reviews lead to high quality science, and so there is some very real society-level benefit that trickles down to the reviewer herself. Maybe.
The culture of anonymous peer review seems totally perverse and broken to me. I’ve heard the argument that anonymity can be important for the situation where Professor AlphaChimp attempts to publish work with severe deficiencies. When Mr. OmegaNerd points those deficiencies out in the manuscript review he can do so without worry of getting bullied by AlphaChimp by virtue of the veil of anonymity. But that same veil prevents the rest of the scientific community from praising Mr. OmegaNerd for his excellent detective work and for having the gumption to communicate about the problems without restraint and with clarity.
Fortunately I think the situation outlined above is less common than many proponents of anonymous peer review would like us to believe. By the time of journal submission many papers are well reasoned. A more common occurrence is that the reviewer, possibly in part due to the lack of incentive to read the paper carefully and write a quality review, writes a careless negative review that prevents the paper from getting published. But probably most common of all is that the reviewers usually provide thoughtful, balanced, and helpful reviews.
While publishing some of my own work recently I found myself with some unusually well thought-out reviews. The comments made by the reviewers would certainly be of interest to future readers of the manuscript. Even though they were anonymous, I wanted to publish the peer reviews along with the paper here on my blog or as comments on the journal’s web site. But after inviting the opinions of several colleagues I decided not to do it. Everyone thought publishing the anonymous reviews was a bad idea, mainly because the reviewers wrote with the expectation that their comments would remain private to the editor, themselves, and the authors. But it also brought my attention to the legal question of copyright on anonymous peer reviews:
Can an author legally publish someone’s anonymous peer review? Do the anonymous reviewers hold an implied copyright on their reviews? If so, and if one were published against a reviewer’s will, would that reviewer have to compromise anonymity to request the removal of a published review? (and induce the streisand effect in doing so)
Even though I wanted to publish the anonymous reviews it seemed unwise to publish them along with the work itself. Had I been able to contact the reviewers to ask permission they might have agreed! But anonymity prevented me from (conveniently) contacting them and the reviewers did not indicate whether I had permission to republish their reviews.
So from now on I have decided to place a copyright statement on all of my reviews to make it abundantly clear to the authors whether they have permission to republish the review. The license I have chosen for my reviews is the Creative Commons Attribution No Derivatives license. That license will allow manuscript authors to publish my review along with their work, so long as the review is published in unmodified form. The attribution aspect implies that my name will be attached to the review. I think that imparts a level of honesty in the review process, and I sign my reviews even if they are negative. I understand not all reviewers share my views on that matter, but even if written anonymously, I think it would be wonderful if reviewers could indicate their willingness for the review to be shared by attaching a copyright licensing statement to their review.
Hear ye reviewers, go forth and write high quality reviews, and let them be published so you can receive credit for your hard work!
It may be possible with some legal gymnastics to attach an enforceable copyright license to an anonymous peer review, but I think that misses the point. Our reviews should be published to encourage high quality review and to provide credit to the reviewers who play an important role in advancing the state of the art in science. Some people argue that such a practice will merely lead to “old boys clubs” or networks of peers who always provide positive reviews of each other’s work and thus compromise the quality of science. But such social networks form whether peer review is anonymous or not. If reviews and reviewer’s names were published then at least there would be some greater transparency and accountability in the social networks. Maybe reviewers should insist on the right to publish their review in exchange for their hard work?
October 14th, 2010 at 3:57 pm
My fear with open (non-anonymous) review is that people would be less likely to give negative feedback on – or even reject – manuscripts of people they know within the community.
Mostly we know at least superficially the others working in the field, and consider some as friends with whom we like to go out for a beer at conferences.
When giving anonymous reports, we can keep on going out for a beer or two occasionally and still provide honest criticism of each other’s work.
October 14th, 2010 at 4:52 pm
Anonymity aside, I still think the issue of review copyright and permissions is very ambiguous. I would love to be able to publish anonymous reviews of my papers, and so it would be great if all reviewers (anonymous or named) would indicate their willingness to have the reviews published with a copyright statement.
That said, people who have fear of writing something negative about a colleagues work need to grow a spine. The several times so far that I’ve signed a negative review it’s turned out pretty well. In one case the authors contacted me and we had a very positive discussion about why I came to the conclusions in the review. If non-anonymous review were the norm most people would acclimate to having a beer with someone who provided a candid evaluation of their work without unleashing a drunken torrent of insults.
October 14th, 2010 at 8:59 pm
Great post and I totally agree. I’ve thought for more than 6 years now that anonymous peer-review is a waste. Basically ever since my first paper in graduate school. The first draft received very thoughtful reviews with a lot of negative comments. Most of those negatives won arguments for me with my advisor (zing!) and one reviewer came up with an experiment idea that worked out very well. They made my first paper a lot better. I still regret that I don’t know who those people are and that they don’t get any credit from readers.
I’d go one step further in peer review reform: I’d require publication of referee comments _alongside_ the originally submitted manuscript. For one thing, this would make referee comments more understandable (presuming the final manuscript fixed the problems). But more importantly, I think it would stop people from submitting unfinished work. Unlike you, I frequently end up thinking that the authors rushed to submission, figuring they’d get a chance to revise and do the work in the meantime. Or they just submit unrefined junk hoping it will slip by. I think they’d stop doing this if they knew their original work would be published.
It seems to me that reasoned arguments like yours to overhaul or abolish pre-publication peer review are ramping up quickly over the past couple years. Maybe it’s time to form a coalition of referees and make demands. This would actually be easy. For example, we get a group of people to agree on some condition, such as: we will only serve as referee if our signed comments are published along side the final manuscript (that’s a weaker demand than I want). of course, most publishers won’t agree to this, but if we stick to our guns: hey, less work for us! Eventually, people will see the wisdom of our way and join us and force change. OK, maybe I’m dreaming, but it seems nice. Actually, what’s preventing me from making that my own ultimatum? Nothing really, my tenure committee would never find out that I was refusing all my referee requests, except for PLoS ONE.
OK, thanks for the great post!
October 14th, 2010 at 9:39 pm
@Steve,
Your suggestion about publishing reviewers comments alongside the article is actually already done in some form by the folks at the journal Biology Direct, http://www.biology-direct.com/
I’ve never published there but as far as I can tell it’s a very limited form of what we’re discussing because reviews generally come from the editorial board members and of course it’s just one of the many thousands of journals out there. Are there other examples?
I’m all for putting together a small movement of reviewers who demand publication of their reviews. Even if there aren’t enough of us to create any real pressure on journal editorial boards we would at least raise awareness about the issue. That will create momentum. Snowball effect. yeah! I propose we start with a website and a short and simple manifesto that supporters can reference. Maybe also gather a set of founding members for the idea and maintain a registration list for supporters.
But one thing still nags me… What to do about the work that goes into reviewing papers that ultimately never get published? Should the reviews be published anyway? And the paper published along with a public REJECTED designation?
October 14th, 2010 at 10:16 pm
@Aaron, yeah that is a problem with rejected papers. One other thing I’d had in mind is that referees would have an option “decline to review” after seeing the manuscript. This “declined to review” would also be published, if they choose to see the MS. So, at least if upon looking it’s a total mess or obviously not suited for the journal, you could choose that option. But I agree, it’s a problem. When I’m in the PLoS ONE frame of mind, it’s less of a problem, because at least you know the paper won’t be declined due to impact assessment. But even there, it’s still a problem. It’d be cool if the original manuscript were automatically posted, but then I think we’ve reached the point of completely post-publication review. Not that that is bad, but I think people are less ready for it at this point. BTW: I’m going to forget to come back here at some point, so you can either holler at me on the friendfeed thread http://friendfeed.com/treeoflife/b26b8131/i-love-this-idea-from-aaron-darling-who-works-in or email
October 15th, 2010 at 11:15 am
[...] Who owns your review? [...]
April 1st, 2012 at 8:11 am
Nice discussion! I believe that review disclosure would help readers to think critically about the work they are reading. The issue of “credit to good reviewers” is also important too, I agree, but could be treated independently from the “copyright” issue. If let to author decide, I guess that people wouldn’t publish alongside their papers bad reviews. Therefore, if we are going in this direction, it would be necessary to be something mandatory not optional, right? Thanks.