Editorial: Lift the veil of secrecy over peer review
ALL’S fair in love and war, they say, but science is supposed to obey more noble ideals. New findings are submitted for publication, the studies are farmed out to experts for objective “peer review” and the best research appears promptly in the most prestigious journals.
Some stem cell biologists are crying foul, however. Last year, 14 researchers in this notoriously competitive field complaining of “unreasonable or obstructive reviews”. The result, they claimed, is that “publication of truly original findings may be delayed or rejected”.
Advertisement
Triggered by this protest, New Scientist scrutinised the dynamics of publication in the most exciting and competitive area of stem cell research, in which cells are “reprogrammed” to acquire the versatility of those of an early-stage embryo. In this fast-moving field, where a Nobel prize is arguably at stake, biologists are racing feverishly to publish their findings in top journals.
Our analysis of more than 200 research papers from 2006 onwards reveals that US-based scientists are enjoying a significant advantage, getting their papers published faster and in more prominent journals (find our data, methods and analyses here). The disparity is likely to spark debate when the (ISSCR) meets in San Francisco next week.
There are several plausible and reasonable explanations, but feelings are running high nonetheless. With two of the most delayed papers coming from a Japanese researcher who pioneered the field, and some of his rivals using controversial channels that give members of the US National Academy of Sciences an inside track to rapid publication, it is easy to see why.
The protest letter called for journals to publish the anonymised comments of researchers who act as reviewers of papers, to expose examples of potential obstruction. Just two of its signatories were from labs in the US. And when leaders of the protest talked to the media, unfair treatment of researchers outside the US was among the complaints. “There does seem to be this bias against groups from the rest of the world,” Robin Lovell-Badge of the UK’s in London told New Scientist.
Research on induced pluripotent stem (iPS) cells is the obvious place to look for biases in publication, given the high stakes involved. One of the signatories of the letter was the pioneer of cellular reprogramming – Shinya Yamanaka of Kyoto University in Japan. Less than four years after he first showed how to reprogram a mouse skin cell, Yamanaka is routinely mentioned as a candidate for a Nobel prize. He may be sharing that honour if other scientists make faster strides towards therapies based on cellular reprogramming. Our reveals that Yamanaka is still the most influential figure in the field, but also shows that several well-connected US-based scientists are giving him a run for his money (see diagram and “The strongest link”).
New Scientist searched the database for studies on iPS cells, recording the dates each was submitted, accepted for publication, and published. Advised by of Emory University in Atlanta, Georgia, whose research employs a branch of statistics called survival analysis, we found that papers submitted by authors outside the US took significantly longer to be accepted and published.
This difference was particularly clear for papers in 29 high-profile journals with an “” of 5 or more (see “What’s the hold-up?”). Impact factor is a measure of the frequency with which a journal’s articles are cited in the scientific literature. We chose this cut-off score to focus on journals that received the protest letter, or those with similar prominence. “It’s really very interesting,” says Lovell-Badge. “I didn’t think it would be possible get quantitative data.”
So what might explain the pattern? Obstruction of papers could happen if a reviewer delays their comments, or makes many demands for changes. The journals with the greatest lag between US and non-US papers deny that their reviews are biased, and say that the former explanation can be ruled out. “The review process itself is quite short, and the majority of the intervening time is taken up by the authors performing revisions to address the criticisms that the reviewers raised,” says Deborah Sweet, editor of , which is the official journal of ISSCR – and is published by Elsevier, a sister company of the publisher of New Scientist.
Bigger, better?
of the Harvard Stem Cell Institute, among the foremost in the field, suggests that leading US labs can deal with revisions more easily because they are often larger and better funded. When asked to run extra experiments, for example, he can quickly deploy junior scientists or hire a commercial lab to do them.
Could it also be that the US-based scientists tend to produce better work? There’s no simple way to measure the calibre of studies in our sample – and if papers are rejected by leading journals and end up lower down the publishing hierarchy, it is hard to separate unfair treatment from genuine differences in quality.
Any deviation from strict merit-based review would bias the entire development of a field. “Papers that are scientifically flawed or comprise only modest technical increments often attract undue profile,” the protest letter argued.
Whatever the reason, US-based researchers were more successful at getting their work into top journals. Overall, our analysis included 216 papers, 119 from scientists working in the US. The sample from higher-impact journals, however, was noticeably skewed, with US-based scientists accounting for 94 of 148 papers.
Poor English is another factor that puts some scientists at a disadvantage. at the Massachusetts Institute of Technology, the leading US-based researcher working on iPS cells, argues that some papers from Asia are so badly written that they are difficult to assess – particularly if they come from a lab with no track record. “There are labs in those countries that are not up to standard,” he asserts. “You get a paper from someone you’ve never heard of, and they’re making outrageous claims.”
“You get a paper from someone you’ve never heard of… making outrageous claims”
If other reviewers are similarly disinclined to trust results from labs they don’t know well, the stronger connections that seem to exist among US-based stem cell biologists may help to explain their advantage.
More mysterious, given his standing in the field, is why two of Yamanaka’s papers were among the 10 with the longest lags. In the most delayed of all, Yamanaka that the tumour-suppressing gene p53 inhibits the formation of iPS cells. The paper took 295 days to be accepted. It was eventually published by Nature in August 2009 alongside four similar studies. “Yamanaka’s paper was submitted months before any of the others,” complains at the University of Cambridge, UK, who coordinated the letter sent to leading journals.
Yamanaka suggests that editors may be less excited by papers from non-US scientists, but may change their minds when they receive similar work from leading labs in the US. In this case, Hochedlinger submitted a paper similar to Yamanaka’s, but nearly six months after him. , Nature‘s chief biology editor, says that each paper is assessed on its own merits. Hochedlinger says he was unaware of Yamanaka’s research on p53 before publication.
“Of papers published in leading journals:
78 per cent from US-based authors accepted within 100 days
54 per cent from authors elsewhere accepted within 100 days”
While arguments about delayed papers rumble on, the data for reveals how some have found a fast track. Each academy member can “contribute” up to four of their papers to PNAS each year, and “communicate” two on behalf of other scientists, in both cases choosing the reviewers. Other academic journals do not have such routes to publication. Of 16 PNAS papers on iPS cells in our sample, 14 came from US-based scientists – and 12 of those went through these routes. The other two were from Japan, one communicated by a Japanese member of the academy.
These numbers are out of proportion to the journal’s overall mix – over the same period, just 37 per cent of PNAS papers were contributed or communicated by academy members. The fact that the iPS cell papers going through these routes were accepted very quickly compared with those in other journals indicates why these inside channels are attractive in such a competitive field. The papers included three from Jaenisch.
The “communicated” channel is due to be retired, and will accept no further submissions from the end of this month. It’s part of a wider effort to “level the playing field”, explains the editor-in-chief of PNAS, , of the University of California, Berkeley. There are no plans to abolish the “contributed” channel.
The protest letter having so far failed to get journals to publish anonymised reviewer comments, arguments about bias will continue. But on one point all of the protagonists can agree. “This whole stem cell field is so overheated… so competitive,” says Jaenisch.
Editorial: Lift the veil of secrecy over peer review
For details of New Scientist‘s data, methods and analyses, go to newscientist.com/article/dn18996
The strongest link
of Kyoto University in Japan is the dominant scientist in cellular reprogramming, but he has stiff competition from a well-linked group of US-based researchers.
To map influence in the field, New Scientist constructed a social network diagram (left) based on citations, the references to each scientist’s work by their peers. Citations are a measure of a researcher’s impact and influence, and are sometimes used to help make decisions on promotions. They can also provide a snapshot of who’s who in a field.
Assisted by Henri Schildt of Imperial College London, a specialist in citation analysis, we looked at references between 148 papers published in prominent journals since 2006 – drawing links where the authors cited one another’s work four or more times.
Yamanaka’s research is referred to by just about everyone. But there are no such links between other scientists outside of the US, and no links to them from any of the US-based researchers. In the US, there is a richer web of connections.
In large part, this reflects the greater number of papers in our sample from scientists in the US. But another tie links the best-connected researchers in the US: the Boston area. Rudolf Jaenisch at the Massachusetts Institute of Technology is Yamanaka’s strongest rival, and two of the other main players – Konrad Hochedlinger and – used to work in his lab. Plath has moved to the University of California, Los Angeles, while Hochedlinger remains nearby at the Harvard Stem Cell Institute, which also hosts the labs of and .
The outsider is at the University of Wisconsin-Madison, who first isolated human embryonic stem cells in 1998. He owes his prominence in this network to winning the race, in a tie with Yamanaka, to make human iPS cells.
