Peer review

Simple stuff, in theory.

Poetry of reality Science

We must know. We will know.

Physics Chemistry Biology Astronomy Geology

A view from the shoulders of giants.

Ocean Lenski affair Photon False placebo Galaxy George Hitchings Chemotherapy Cambridge University Press v - t - e

Peer review is the process of subjecting scholarly work to review by other experts in the field. The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers" (i.e., other experts in the same field) to assess the quality of the work. A similar approach is generally taken to evaluate research proposals submitted to agencies for funding, such as the National Science Foundation (U.S.) or Natural Sciences and Engineering Research Council (Canada), where the proposals are sent out to qualified scientists to assess whether the proposed projects merit funding.

Initial and ongoing peer review[edit]

The first and most notable step in peer review is when a piece of scientific work is looked at by reviewers (sometimes called "referees") for approval before publication. Typically each paper is evaluated by three reviewers. The journal's editor then evaluates the reviews and decides whether the paper should be published or rejected. Acceptance usually is provisional on the authors making revisions to take the reviewers' comments into account.

Approval by peer review is a necessary but not sufficient criterion for quality work. It mostly judges basic competence, such as minimizing the chances that pseudoscience can masquerade as science, and filters out trivial or low-quality work that would not contribute anything of value.

Following publication, peer review is an ongoing process where work is open to scrutiny by the scientific community — this is what publication allows a scientist to do. The process is designed to ensure that the work meets the standards of the field in question and science in general. A paper that survives initial peer review may be considered rubbish when examined more widely after publication.

Selection of peer reviewers[edit]

Being a peer reviewer is often a thankless task; one does not usually get paid for it, nor does one get much recognition except perhaps from the journal editor.^[1] Nonetheless, scientists usually perform peer reviews when asked to do so. This is because it is considered a service to the profession:^[1] if I don't review my colleague's paper, who will review mine? Generally speaking, journal editors will seek out peer reviewers who:^[1][2]

• are actively publishing in the field,
• have expertise relevant to the paper to be reviewed,
• and can give an independent review (i.e., no financial conflicts and not affiliated with the authors).

Peer reviews are sometimes double-blind;^[2] the reviewers do not know who the authors are, and the authors do not know who the reviewers are; only the journal editor knows. This helps to assure that the review is independent. The traditional method is single blind review, where the reviewer can see the author's identity, but not vice versa. A newly introduced style is open peer review, where the reviewers' identities are published along with the article.

Sometimes the level of expertise required to review the paper is so specific that the authors can guess who the reviewers are or vice versa. One method that can sometimes be used to address this issue is finding reviewers with expertise in different aspects of the paper (e.g., one for statistics and one for medicine).

Relevance to the scientific community[edit]

Peer review is a key part of the scientific method, where the system's goal is to ensure that work is stripped of biases, unjustified assumptions, and other errors through the review by one's professional colleagues. Accordingly, peer in this context implies equals: i.e., the reviewers or judges should have the same or reasonably similar qualifications that the work's author has — or claims to have. No ideology, other than the commitment to "rigorous empiricism… without which no man is a scientist," matters in asking whether a person is a scientific "peer".^[3]:42

Thomas Kuhn, in the process of seeking to define the nature of science and the nature of "scientific progress", says in The Structure of Scientific Revolutions that the unique composition of the scientific community as composed of disinterested (read: unbiased) intellectual equals, is alone capable of generating scientific "progress."^[3]:168 Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life… the prohibition of appeals to heads of state or to the populace at large in matters scientific."^[3] Politics has no place in science, and the goal of peer review is to strip politics out of the equation.

More prestigious journals have a more thorough peer-review process and have tighter criteria for the level of work they can publish — indeed, this is why they are prestigious, as only the most groundbreaking work is published in these journals. The two top journals in the world for science are Science and Nature, and getting a paper in them is considered a great achievement.^{[note 1]} The vetting process for these two journals is intense, with less than 10% of submitted manuscripts going forward for publication^[4] — that's 10% of the manuscripts scientists think are good enough. These top publications also tend to "coach" their submissions, improving the paper's readability, conciseness, and clarity (again, this is why they're considered the best).^[5] This isn't to belittle "lesser" journals at all; the publishers of Science, for example, openly acknowledge that articles may be rejected "not because the science isn't spectacular but because the area is no longer 'hot.'"^[6] More typical journals in scientific fields ultimately accept perhaps 50-70% of submissions, though some revisions will almost always be required before publication.

Possible problems[edit]

Like all human activities, peer review can be subject to biases in certain situations or if insufficient care is taken in the selection of reviewers:

1. In very competitive areas of sciences, reviewers may be tempted to gain an unfair advantage when they receive a rival's work for review long before publication and may even try to scoop the results. Good editors will be aware of the potential for such conflicts and consider them when making their final judgment on accepting a paper.
2. In the case of competing schools of thought, some reviewers might be influenced by their adherence to a certain point of view and give negative reviews to colleagues from an opposing school of thought. Again, the journal's editor should be able to judge this issue.
3. If the field of experts for a given topic is small, there is a certain likelihood that the reviewer may have a relationship of animosity, rivalry, or friendship with the author that biases the review process.
4. The peer-review process doesn't involve replicating experiments or studies to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work is suitable for publication as it is presented. Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
5. In journals lacking real oversight from publishers, editorial board members may be able to publish their own articles by either bypassing the process or hand-picking their own peer reviewers. This created a small scandal in the mathematics community when Chaos, Solitons and Fractals was found to have carried hundreds of articles by its editor-in-chief, Mohamed El Naschie.
6. The editors of many journals can reject papers before they reach the peer review process.^[7][8] While this can be good in that it avoids wasting reviewers' time on obvious junk, depending on the rules, this could lead to the unjustified dismissal of good science.
7. In 2013 and 2014, the editors of the Journal of Vibration and Control and SAGE uncovered a peer-review ring involving assumed and fabricated identities, which led to the retraction of 60 published articles.^[9] As of January 2015, the number of fake-peer review articles exceeded 100.^[10] The individual record holder for retractions is currently "Yoshitaka Fujii, a Japanese anesthesiologist who fabricated his findings in at least 183 papers".^[11][12]

In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the journal's editors. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with the publication, as the article or paper remains available for all interested parties to view — if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review process remains the most objective and qualified way to assess scientific work that has ever been developed.

Peer review as a legal benchmark[edit]

Without a doubt, the twentieth century saw an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove a scientific argument to win his case, the court must have a way of judging the science.

Courts have met this challenge by adopting Federal Rule of Evidence 702 and the Daubert test, which require that "expert testimony" — a category that encompasses scientific testimony — be methodologically sound.^[13] In law student's shorthand, the Court in a Daubert inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer-reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the Daubert court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science and agreed that the only cure for bad science is vigorous "cross examination."^[14]

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review before science can be science.

What peer review is not[edit]

Many people think that the process of peer review is meant to settle the actual validity of the work and that in any paper that has passed peer review, the science is entirely correct. This is not the case. Peer review is an "entry-level" sort of test that weeds out the pseudoscience and obviously bad work but is not intended to be a catch-all for outright fraud or experimental error — reviewers simply challenge the rigor by which scientists are reporting their own work or challenge their conclusions if they haven't successfully eliminated competing hypotheses. Often enough, the demand for the right data and better conclusions made by reviewers is more than enough to ensure the work is valid enough, as the process is about making sure everything is submitted and out in the open with nothing hidden. Due to this, direct replication and validation aren't usually a priority or even a necessity for peer review.

There are a few exceptions. For example, the American Chemistry Society won't accept computational chemistry papers unless the results have been verified. This is because taking parameters from a paper's supplemental material and running it on a computer for a few hours is practical; rigorously replicating experiments that may have taken months to get right and require specialized and bespoke equipment is not.

Passing peer review and publication is indicative that (by the standards of the journal in question) the science is thorough, there are no glaring omissions, and the interpretation of the results presented is at least plausible, but this does not cement the science. Further publications and research can then use the data contained in the paper, and its conclusions can be amended (in worst-case scenarios, retracted) in later publications.^[15]

To make a legal analogy, if it is erroneously assumed that the peer review process is like a trial (the case is either proven true or dismissed), the actual process is more like an arraignment, only verifying that the case has enough merit to be heard. Indeed the "trial" part of scientific work is an ongoing and continuous process as other scientists cite the paper or attempt to replicate or use it in their own work.

It is also worth noting who those "peers" may be, as practitioners of pseudoscience might form a circle of pseudoscientists who start a pseudoscientific journal. It isn't the support of a claim that makes it true; it's the honest attempts to disprove a claim through experimentation that solidifies it.

The problem with not peer reviewing[edit]

Journals that do not have a peer-review process can give a veneer of respectability to otherwise bad science. Such publications can be picked up by the popular media and be a source for spreading irrationality, and even be responsible for causing hundreds of thousands of deaths. Such was the case with the journal Medical Hypotheses (published by Elsevier), which was not peer-reviewed before June 2010. The journal had published papers on AIDS denialism^[16] and was the source of the thiomersal anti-vaccine hysteria, "the most damaging medical hoax of the last 100 years."^[17]

Creationists and peer review[edit]

Allegations of bias in peer review[edit]

Creationists, whose "theories" have repeatedly been rejected by peer-reviewed journals,^{[note 2]} claim that the system is rigged against them since theistic explanations for natural phenomena are rejected out of hand because scientists assume methodological naturalism when working. The argument runs something like this: institutions of peer review assume that theistic explanations are invalid since a built-in requirement of naturalism constitutes atheistic bias.^[18]

While prima facie is compelling and probably enough to dupe a few uninformed bystanders, this argument does not hold up to logic. The purpose of science (and rationalism) is to determine explanations of natural phenomena, with reference only to objective natural indicia, to create a body of knowledge that all of humanity can use and apply, if possible, in engineering and medicine. The purposes of science — to determine objective truth, free of a precondition of religious belief, and to create usable knowledge — are incompatible with theistic explanations.

Of course, naturalism has limits — if no objective explanation of a phenomenon exists, perhaps some theistic agency is at work. Thus, naturalism does not displace theistic explanations; rather, it pushes back theistic explanations where they are incomplete and inaccurate. To argue that theism should be a start for science then completely discounts the nature of science: accusations of bias fall apart accordingly.

Attempts by creationists to intelligently design peer review[edit]

A common criticism of creationism and its intellectual kindred (baraminology, intelligent design, and creation science) is that these "sciences" can never pass peer review, as they fail to conform to scientific methodology. In addition to accusations of bias, disproven above, creationists have attempted to create their own "peer review publications" in a vain effort to create their own body of "scientific" knowledge and render neutral the "no peer review" criticism.

First as tragedy: the creation science "mainstream"[edit]

Of course, the "symptom" that creationism lacks peer review is indicative of the "condition" that it, and its brethren, lack real scientific merit: thus, by creating new "peer review" periodicals, "cdesign proponentsists" merely cure the symptom, not the underlying condition, i.e., the unscientific nature of their theology. In short, creationism does not gain merit by having an empty label ("Now Peer Reviewed!") attached to it.

However, several creationist groups have not gotten the memo: they continue to attempt to create sham peer review journals, each of which betrays the ultimate failing of their own discipline by failing to conform to the methodology of peer review and merely usurping the label. Several examples come to mind: Answers in Genesis, CreationWiki, JPANDS, and more "mainstream" creation scientists.

As one commentator has noted, creation "science" under its infantile peer review system approximates the level of depth of inquiry of 18^th-century science.^[19] Since creation science cannot be tested without failing horribly (when it can even be tested), its intellectual discourse resembles, essentially, the creation of wild guesses, the discussion of said guesses around a fireplace, the assembly of these guesses in journals, the commitment to investigate them at some point, and then, the break for dinner. The process then repeats. The system entails no accountability because, by its nature, it cannot be accountable.

Then as farce: CreationWiki[edit]

CreationWiki, "a free encyclopedia of apologetics that is being assembled by an international team of missionaries,"^[20] has attempted to create a peer review system^[21] to vet CreationWiki articles, assuming that the result is something of "peer reviewed" quality. The premise is either deliberate deceit or unconscious misapplication of a scientific term of art; since science depends upon a group of "peers," those being individuals versed in high sciences, not apologetics, nothing produced by CreationWiki could ever approximate peer review unless its membership base changed by nearly 100%.

Evidence suggests that the conflation of "peer review" with "casual editing" by CreationWiki indicates a larger confusion, within creationist circles, about the meaning of "peer review." CreationWiki administrators, in particular, seem to believe that "peer review" is meant to be deliberately slanted: in their own words, "that is the goal of peer reviews in general — to uphold the consensus position."^{[note 3]} Apparently, creationists reconcile their own exclusion from academic journals not by conceding their own lack of merit but by asserting that peer review, in general, is meant to "censor" and responding in kind: in their own words, "peer review is done by professionals who hold to a POV and censor other views."^[23] In short, creationists see the denial of publication for lack of scientific merit as censorship for point of view (see balance fallacy): a rhetorical trick often used in creationist literature, as in Expelled: No Intelligence Allowed, to make the cause out as a pariah.

Latecomers: Answers in Genesis[edit]

In 2008, Answers in Genesis attempted to found its own "academic" journal, Answers Research Journal.^[24] Hilarity continues to accrue at a fairly rapid rate.

What peer review looks like[edit]

For an example of what real peer review looks like, see the peer review section of Clair, Carole, et al.'s "Dose-dependent positive association between cigarette smoking, abdominal obesity and body fat", published in BioMed Central Public Health in 2011:^[25]

Notice that:

• The paper was reviewed 6 separate times over 11 months.
• The paper was rejected 5 separate times; the author revised it each time.
• The full peer review history and each of the 6 reviewer reports are publicly available
• The paper was reviewed by two qualified individuals from wholly unaffiliated universities^[26][27]

The humanities[edit]

Fields in the humanities, such as history, literary criticism, philosophy, art history, cultural anthropology, psychology, women's studies, African-American studies, classics, and poetics, differ from the hard sciences regarding methodology and scholarly practice. In these cases, peer review exists to see if they hold to the academic and methodological standards of the specific disciplines, if they are internally consistent, if they engage with the established scholarship on the topic, if they account for long-existing problems, and if they make a meaningful contribution to the scholarship.

See also[edit]

• Bentham Science Publishers
• Vanity publishing
• Diederik Stapel‎
• Publication bias

External links[edit]

• Jennifer Raff — How to read and understand a scientific paper: a guide for non-scientists
• Pro: Retraction Watch — a searchable database of retracted published papers
• Con: SCIgen - An Automatic CS Paper Generator

Notes[edit]

References[edit]