Tuesday, November 10, 2015

Peersnickety Review: Rant on My Recent Battle With Peer Reviewers

I'd like to relate a tale of exasperation with the peer review process that I recently experienced and that is probably all too familiar - but one that most folks are too timid to complain publicly about.

Nevermind that laypersons think that peer review means that your peers are reviewing your actual data for accuracy and fidelity (they are not, they are reviewing only your manuscript, final analyses, and conclusions), which causes them to be perplexed when revelations of fraudulent data published in top journals are reported.  Nevermind that the website Retraction Watch, which began as a small side project now has daily and twice daily postings of retracted papers.  Nevermind that some scientists have built entire careers on faked data.  Nevermind that the fact that something has been peer reviewed provides very little in the way of assurance that the report contains anything other than rubbish.  Nevermind that leading investigators publish the same reviews over and over in different journals with the same figures and sometimes the same text.

The entire process is cumbersome, time consuming, frustrating, and of dubious value as currently practiced.

Last year I was invited by the editors of Chest to write a "contemporary review of ionized calcium in the ICU - should it be measured?  should it be treated?"  I am not aware of why I was selected for this, but I infer that someone suggested me as the author because of my prior research in medical decision making and because of the monograph we wrote several years back called Laboratory Testing in the ICU which applied principles of rational decision making such as Bayesian methods and back-of-the-envelope cost benefit analyses to make a framework of rational laboratory testing in the ICU.  I accepted the invitation, even knowing it would entail a good deal of work for me that would be entirely uncompensated, save for buttressing my fragile ego, he said allegorically.

Now, consider for an instant the extra barriers that I, as a non-academic physician faced in agreeing to do this.  As a non-academic physician, I do not have access to a medical library, and of course the Chest editors do not have a way to grant me access.  That is, non-academic physicians doing scholarly work such as this are effectively disenfranchised from the infrastructure that they need to do scholarly work.  Fortunately for me, my wife was a student at the University of Utah during this time so I was able to access the University library with her help.  Whether academic centers and peer-reviewed journals ought to have a monopoly on this information is a matter for debate elsewhere, and not a trivial one.

Next, consider the opportunity costs for me.  This paper required approximately ten to twenty hours work per week on and off for over six months.  That is a lot of time.  If I were an academic physician trying to pad my resume or show "productivity" or ingratiate myself to some superior or impress some juniors, it would be somewhat easy to justify that time (even though non-original articles such as reviews, perhaps deservedly, don't enjoy much clout), and, in the end, I would be being paid for it (even if it were a paltry salary - that's a gripe the academicians ought to blog about).  Moreover, I would not have had to replace my computer and my software to update them for the reference management - at my own expense.

Alas, I reviewed the admittedly meager literature exhsaustively, summarized it, applied principles of rational decision making to the questions I was asked to address in light of the paltry evidence available, and submitted the paper.  When you submit the paper, you have the opportunity to suggest reviewers (makes the editor's job easier) and to suggest people you do not want to review the paper (your intellectual enemies, I suppose.)  Two months ago I received the responses of my sometimes peersnickety reviewers.

Here's a structural problem with peer review:  I don't know who the reviewers are or what institution they're with, but they know who I am and what institution I'm affiliated with (some hospitals in Utah they have never heard of).  The justification for this imbalance is not obvious and difficult to imagine.  Either the paper should be reviewed anonymously - anonymity should be bilateral - or the author(s) and the reviewers ought to know who each other are.

Reviewer 2 is generally favorable in his/her comments, but wants me to add some stuff about animal experiments (why would I do that in a practical review?) - I decline.

Reviewer 1 wants me to include in the text and bibliography this 2014 JAMA article about Vitamin D.  I was asked to write an article about ionized calcium (iCa) and whether it should be measured and treated.  They did not even measure calcium levels in the JAMA article, so why would I include it?  It is simply not relevant to my review.  I could choose to placate and appease the reviewer and find a way to work it in, but I'm at my editorially imposed word limit, and quite simply it's not germane so it's not going in.

Reviewer 1 also wants me to speculate at greater length than I already have the evolutionary forces that are driving changes in iCa levels in critical illness.  But reviewer 3, the most persnickety of all, thinks that I've already been too speculative about that notion.  How the heck am I to resolve this conflict?  By not changing anything, that's how.

Several reviewers also imply that I should have spent more time discussing the physiology of calcium homeostasis.  I was not asked to write a review on calcium homeostasis, I was asked, in the invitation, to discuss the practical questions: should we measure and treat abnormal iCa levels, which is what I did.  I am not now, after writing the entire paper, going to go back and rewrite it on the post hoc whims of two anonymous reviewers who were either not provided with my original invitation letter or who ignored the task I was charged with by the editors who invited it.

Then there is number 3.  Oh, reviewer number 3.  Number 3 says that "the entire paper will require extensive revision because of numerous run on sentences."  If there is one thing I know how to do as well as anybody else, it is write, and my grammar is generally impeccable (if arguably verbose - I'm working on this).  So this reviewer just doesn't like my writing style, and I don't like his/her lack of specificity with language.  So, I tell him(her):
A run on sentence is one in which two sentences or two independent clauses are joined without an appropriate conjunction or punctuation, and it constitutes a grammatical error.  I am confident that there are no grammatical errors in the manuscript.  Complex sentence structure is a stylistic flourish characteristic of this author’s writing which some readers will subjectively rate more or less favorably than others.  I respectfully disagree that extensive revision is required in the absence of grammatical errors.
Now what I was supposed to do, as taught over the years by several sage mentors and advisers, was to suck up to the reviewers, thank them graciously for all their comments and suggestions, not argue with them, and make as many changes as I could to appease and please them.

But I didn't do that and I'm not going to do that.  Because I wrote this article in earnest without compensation and at some financial and a lot of lost opportunity expense to me; because the peersnickety reviewers made few if any substantive comments or pointed out any significant inaccuracies; because I'm not going to make major revisions based on reviewer whims or wishes after the fact about what the editors should have asked me to write about; and because if reviewer 3 wants to go lengua a lengua about grammar and language, perhaps s/he should reveal his anonymous self and we can discuss the finer points of grammar and run on sentences - here or elsewhere.

Meanwhile, the editors of Chest have some work to do.  It is not my task to shepherd and police their wayward reviewers.  If the reviewers disagree or fail to offer suggestions for substantive changes, they editors ought to referee and use their own judgment as to who is being reasonable and who is not.  And if they don't like the pro bono work I did for them, that's fine with me.  I'll not be a victim of sunk cost bias.  I'm perfectly happy to cut bait with Chest and pay a couple hundred dollars to have it published in some garbage journal, so at least it will have a pubmed ID and be in the public domain (where is which most research ought to be).

Or I'll just publish it here on the blog, for free, peer reviewed by me and the comments that follow the post.  Stay tuned.

Edited 11/24/25:  It appears that my strategem worked.  I received the editor's response today, and they wish for me to make a few very small and reasonable revisions.  Not "extensive revisions" for "run on sentences".

There are two lessons for me here - one is that if you don't feel beholden to these reviewers, and editors, and this process, and you are willing to lose or delay your publication and incur more work submitting it elsewhere, you don't have to tolerate peersnickety review.  But if you're gunning for promotions, may the Lord help you, you're taking a gamble.

Two is that this whole process is treated as some sort of game, like "Oh, I'm asked to review this paper written by this Aberegg, guy, who's he?  It's bloodsport time, I'll nitpick him to madness, just like the last reviewer did to me!"  That is complete nonsense.  I applaud the editors of Chest for finally intervening and putting a stop to it.  Other editors should follow suit.  A culture change is in order.


  1. I look forward to hearing the response from the editor.

    1. Yes, it's become entertainment for me too. To quote a wise man I knew in Baltimore - "you don't own something until you don't care if you lose it."

  2. i read your posts with zeal. I am a noobie trying to break into the academic world, and I can already sense the frustrations involved. Thank you for speaking out.

  3. I updated in italics at the bottom of the post the most recent development - basically acceptance. I will link to the paper when it comes out, hopefully y'all won't mind the "run on sentences"!

  4. On one hand, you have an invited review in Chest - not bad. You have to feel for the reviewers on the other side, however. Although you're not getting paid for your efforts, at least you're getting a pub. The hapless reviewer is usually an overburdened low-ranking academic trying to stay afloat who now gets an Aberegg manuscript on iCa2+ in his/her in-box. Reward for that? Oh - a thank you at the end of the year in the back of the journal. Joy - I'll heat my house with its burning embers. Not to say I don't sympathize with you - I think reviewers always feel like they have to say something, otherwise it makes them look like they haven't read the manuscript. And yes, some of us are bitter and angry ;)

  5. I can't disagree with any of that. I too am a reviewer, another thankless job I have. (Although ATS is trying - trying - to establish a system by which you can receive CME for your efforts. This is a good step forward.) Interestingly, of the dozen or so manuscripts I have recently reviewed, I have recommended most be rejected, for what I thought were fair and cogent reasons. So, ironically, I'm a hardass reviewer too!