Problem with peer review

(Redirected from Peer review)
Jump to navigation Jump to search

This article by Dr Piotr Wozniak is part of SuperMemo Guru series on memory, learning, creativity, and problem solving.

What is peer review?

Peer review is the most popular approach to assuring high quality and reliability of scientific publications.

Peer review is considered a cornerstone of reliable science. It helps scientists read all papers in the field in a hurry. However, peer review also eliminates more inspiring ideas, i.e. those that are most controversial. Those that require extra brain effort to make a judgement on their plausibility.

I am a great fan of a different kind of peer review. I love when my colleagues provide meaningful criticism that helps me improve my message. I always learn something new. However, what is properly called "peer review" in science, I call "peer censorship" that limits author's freedom and independence. Peer review improves quality. Peer censorship blocks creativity.

Peer review curbs freedom and creativity

I see peer review as a limitation on one's creative freedom. It is the same kind of submission that occurs at school. It is hard to accept that someone is supposed to be always smarter than you, and have the final say on your own ideas. Someone can degrade your efforts and send you to a second rank journal, or delay your progress for ever.

I hear a lot of complaints about peer review from my colleagues. It so happens that the loudest complaints come from young, rebellious, and highly creative scientists. They want to change the world, but the old guard stands in their way.

False beliefs are omnipresent, but there is value to wrong models. In the long-term, we inevitably converge on the truth.

Power of blogging

I enjoy working at SuperMemo Guru primarily for its creative freedom aspect. If I have an idea today, someone can read it minutes later, and soak inspiration. This is how ideas should spread: at the speed of light. The lag of months introduced by peer review slows down collective creativity. Errors in thinking can be inspirational too. Having creative fun with words and terminology can make ideas more entertaining. This also makes it easier for them to be understood and passed forward.

Peer review increases scientific standards, but it also dramatically decreases chances of learning something unique and wildly inspirational.

When great scientists blog about their own research, the reading is captivating, and inspiration tripled

Combating fake news

There are hundreds of quacks and pseudoscientists with degrees who freely use societal naivete to mislead the public while earning a pretty penny in the process. They write books, and use social media to spread false memes while promoting their own published works. When a well-schooled scientists is limited to peer review, he probably does not pay attention. If he occasionally finds out about the world of falsity that pulsates on the net, he may only sigh in helplessness. Peer review is not concerned with misconceptions, myths, quackery and sheer lies of the blogosphere. This is why the only way to combat fake prophets is to engage in the war of blogs. Sadly, the velocity of falsity vectors depends on their popular appeal. For example, it is very easy to convince parents that digital media are a threat to intelligence. Many parents already suspect this to be true as they see a drop in school performance. On the other hand, when I encourage freedom in learning, which includes freedom to play videogames, I am either saying what a great deal of parents already know, or what goes against the intuition of those who see electronics as a threat. My message is useless for most parents: either obvious or hard to comprehend. My words are most useful for children who need to combat authoritarian forces.

Here are some examples that would never qualify as serious research papers:


The web will spark a revolution in peer review. It will advance towards a more open science. There is nothing wrong with keeping the process slow as long as we can have a chance to tap to early ideas. They can be poorly worded or wrong. Everyone could choose his filters: either (a) Give me only clean and pure material after months of peer review, or (b) Give me any inspiration I can get in the topic please! There are scientific brains out there that keep spewing gold even while writing in semi-comatose state late in the night over e-mail. I would rather read those great half-baked ideas that dig into well-polished peer reviewed material of lesser authors. In case of "boring peers", the perfection of form may lure the reader into processing reams of formal texts with little actual quality content.

Freedom of expression is the best ally of expression itself. Freedom powers the creative pleasures, which act as a reward that sparks further creativity. Peer review is inhibitory. Many young scientists hate and quit science because of the pressure of publish or perish. I saw that creative block and displeasure develop in very talented people. It is the same mechanism of learned helplessness that suppresses the learn drive and creative thinking at school and in science.

Stephen Wolfram

I am a great fan of Stephen Wolfram. This is the new type of scientist with a potential to change the world. He is beautifully arrogant. This is typical of people with strong models that race ahead of their time. Wolfram is not part of the academic mill and can ignore the peer review. If my words have little weight for you, take it from the genius mind himself (in reference to "A New Kind of Science"):

A typical issue that came up was how the book was vetted or checked. In academia, there’s the idea that “peer review” is the ultimate method of checking anything. And perhaps in a world where everyone has infinite time, and nobody operates according to their own self-interest, this might be true. But in reality, peer review is fraught with error, often quite corrupt, and even in the best case strongly biased toward avoiding new ideas and maintaining the status quo

Youth and creativity

One of my young friends, a budding and an extremely talented scientist, wrote to me recently:

If for some reason I won enough money not to ever have to work again, I would leave the academia, I would not bother to have done a PhD, and I would still do science. I care about the academic world only for one reason: I get paid. Except for that, I don't care and I want out. Just think about it, I could do whatever I want all day long. And I would not have to take silly courses, or to submit articles to incompetent peer-reviewers and editors

First ever publication on spaced repetition

Ralph Merkle invented Public Key Cryptography in 1974. He was an undergraduate student. His texts are clear, convincing and brilliant. However, his supervisors did not see the breakthrough. Some time later, Merkle attempted to publish a paper, but he hit the wall of peer review. Reviewers expected him to provide well referenced text with all prior research in the field listed. He was rejected. An anonymous reviewer guilty of robbing a young man of a breakthrough publication has never been identified. A breakthrough idea was dismissed with Experience shows that it is extremely dangerous to transmit key information in the clear. No kidding! It is as if someone said: Spaced repetition makes little sense. Experience shows that we forget eventually.

Publication of Merkle's paper dragged for 3 long years during which others have published their ideas that later clamored for precedence.

Merkle's case shows uncanny resemblance to my own efforts with Dr Edward Gorzelanczyk back in 1991-1994. The similarity is so close that if you take Merkle's words, and replace Public Key Cryptography with spaced repetition (the idea), you will get our own story word for word:

With a blanket rejection of the idea by an "expert", the Editor rejected my article. She "was particularly bothered by the fact that there are no references to the literature. Has anyone else ever investigated this approach. If they consider it and reject it, why?"
I had failed to provide any references to the prior work, and the reasons previous workers in the field had rejected it as impossible. The term for the idea did not yet exist, and there were no previous workers in the field. There were no words for what I had done, and looking up a concept to show that no one had previously thought of it is difficult. This is not a unique problem: it illustrates a problem faced by anyone trying to explain a new idea to an "expert" who expects a properly referenced article anytime anyone tries to explain something to them. The more a new idea is unrelated to any prior idea or concept the more it must appear as a squawling bastard, naked and alone, appearing de novo and lacking any respectable pedigree or family to vouch for its acceptability

30 years later, Merkle found the original student project in his archives (see PDF original). This reminds me of my accidental discovery of my first forgetting curve graph. I plotted it casually in 1984, and found by accident in Spring 2018 while working over History of spaced repetition.

I was fortunate in one respect. There have been no attempts to publish an effective algorithm for computing optimum intervals in spaced repetition for decades. For that one reason, our 1994 publication will rather have its cozy place in history. Unfortunately, this is also a reflection of poor demand and poor social invention. Public Key Cryptography is an industry today. Spaced repetition is still largely unknown despite an adoption going into millions.

Bless Wikipedia, bless Google, bless the web. If I have an idea today, I learn the ABC at Wikipedia, Google for precedence or concurrence, and publish the idea the next day here. Peer review don't bother me any more.

Seminal Rat Park

The Rat Park experiment changed our view of addiction.

While writing about Rat park, one of young skeptical writers hinted that "Despite his claims of a revolutionary breakthrough, Alexander had trouble finding a journal to publish his results. Both Science and Nature rejected the study for publication, likely due to significant problems with the methodology and results" (source). I had problems with being accepted too (Science, Memory and Cognition, etc.), and this made me ultimately give up on peer review. I prefer the creative outlet of SuperMemo Guru over a combat with people who I do not know and who do not seem to understand what I am writing about. What is worst, kind suggestions often make the text even worse. You cannot add a lighthearted comment or a metaphor that would help the reader.

"Problems" with peer review are indeed "likely" in cases where the reviewers know nothing of the plausibility of research outcomes. However, rejection is a very poor criterion for evaluating the outcomes, esp. when the research is novel, or multidisciplinary, or goes against the established paradigm. My own spaced repetition or two component model of long-term memory are good examples. Trouble with rejection only made Bruce Alexander more of a hero for me.

See also

For more texts on memory, learning, sleep, creativity, and problem solving, see Super Memory Guru