Bohannon’s Science Sting – playing devil’s advocate and proposing a solution

[Update: I realise that perhaps many of you are not going to have the same perspective about the Science Sting that I have purposefully taken here (hence the title). Apologies in advance.]

It took a journalist (albeit one with a PhD in molecular biology) to reveal the extent of the peer-review problems among predatory journals. Bohannon submitted fatally flawed and boringly bad articles to a set of open access journals that charge fees to publish. Of 255 submissions with results, 157 journals (62%) accepted the article. This did not come as a surprise to a lot of people who have been watching or interacting with this vast underworld of predatory publishing.

Simply, fee-charging predatory open access publishers make more money if they accept more articles. It’s not going to be good for business in the long-term (because of reputation problems) but it seems to be working quite well for a number of publishers right now. The need/desire for profits is also a big problem in subscription journals but often for very different reasons (where profit relies on denying access).

But is it working for them? There is limited evidence to suggest that business is good for predatory journals. Hindawi (previously listed as borderline predatory) apparently makes 52% profits, which is phenomenal compared to the already obscene 36% operating profits reported by Elsevier. The reason why business is so good is the direct consequence of the publish or perish mentality that pervades academia. Add to that the hundreds of thousands of students graduating with PhDs each year, the low overheads associated with starting a journal, and paid gold open access becomes an attractive business proposition for any shonky operator.

Response from those heavily invested in open access

Bohannon tells us that he talked about the work with “a small group of scientists who care deeply about open access”, and in the news article explained carefully that the growth of open access has multiplied the problem of predatory journals. This is because predatory publishers are much more likely to opt for a paid gold open access model. I don’t think anyone can disagree with that. The problem is that people who care deeply about open access appear to have often interpreted the news article in a defensive way because of their particular perspectives on open access. There’s a whole bunch of responses to the article that come from the full spectrum of open access advocates and I have made a list below.

To play the devil’s advocate, here’s what they have often missed in their responses.

  • Bohannon correctly included the suggestion that targeting the low end of subscription journals could produce the same result and explicitly indicated that he did not examine them.
  • The article is directed at a particular subset of journals that charge fees and are listed on the DOAJ or on Beall’s list. The aims match the selection of journals and both are clear. At no point does Bohannon say that the chosen set represents open access generally.
  • It is an article about peer review. The fact that the targeted journals were all paid gold open access journals is important to note but that point has very little to do with the huge problems in peer review, which we all know are pervasive and a bigger problem in gold open access.

The results also show that Beall’s list does a reasonably good job of compiling the problems. It was nice to see that Beall was (mostly) vindicated in his identification of dodgy publishers. Over 80% of the journals on Beall’s list accepted the article after some kind of a review. For journals on DOAJ, the proportion was 45%. Much lower but still substantial.

It was unsurprising that Beall’s list produced the highest proportion of accepted articles and the highest proportion of missing peer review, that DOAJ produced fewer in each of the two. Glass-half-full types would have focused more heavily on this to show that there were in fact plenty of good open access journals that do charge fees, do undertake peer review or reject on first principles, and noted that even among a targeted group of journals accused of predatory behaviour, some still undertook peer review.

So what?

There are two important things we need to remember when interpreting the results of the Bohannon’s Science Sting. Firstly, predatory journals and open access journals are not synonymous but there are a lot of predatory journals that are open access. Bohannon did not conflate them in his report and explained his methods better than most journal articles I’ve seen recently. Secondly, if open access is to eventually replace the subscription model completely, then the people that are best placed to tackle predatory journals using open access models are the people that are already measuring, curating, listing and analysing open access. Houses in order, so to speak.

DOAJ should be the first to act – directly with the journals that accepted the paper, and then with the wider group of fee-charging journals. It doesn’t matter what proportion of subscription articles *would* have accepted the papers, the problem still exists no matter how much the “lack of a proper control” or “Science conspiracy against open access” arguments get thrown around to dilute the message.

Solving the problem

If you want a radical solution to the problem of bad peer review in predatory journals, there’s an obvious one that no one seems to be suggesting:

Remove profits from publishing entirely.

The method is simple. Only index and recognise articles that are published with non-profit and flat-rate subsidised (platinum open access) journals – like some of those published by the various scientific societies. Don’t pay journals for “how much” but for “how well”. If subscription journals and for-profit open access journals can’t be cited, indexed, or contribute to career progression, then the market for predatory journals disappears.

What others have been saying

Here is a list of other responses to Bohannon’s Science Sting.

1. Graham Steel: The Publishing “Sting”, the reaction, and the outcome

2. Åse Innes-Ker: A publishing sting, but what was stung?

3. Björn Brembs: Science Magazine Rejects Data, Publishes Anecdote

4. Claire Shaw: Hundreds of open access journals accept fake science paper

5. Peter Suber: New “sting” of weak open-access journals.

6. Curt Rice: What Science — and the Gonzo Scientist — got wrong: open access will make research better

7. Martin Eve: Flawed sting operation singles out open access journals  – and a longer version

8. Ernesto Priego: Who’s Afraid of Open Access?

9. Nigel Hawkes: Spoof research paper is accepted by 157 journals

10. Michael Eisen: I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals

11. Mike Taylor (SV-POW): John Bohannon’s peer-review sting against Science

12. Fabiana Kubke (makes good points about navigating open access): Science gone bad

13. Lenny Teytelman (interesting take & new analysis): What hurts science – rejection of good or acceptance of bad?

14. The Directory of Open Access Journals: DOAJ’s response to the recent article in Science entitled “Who’s Afraid of Peer Review?”

15. Open Access Scholarly Publishers Association: OASPA’s response to the recent article in Science entitled “Who’s Afraid of Peer Review?”

16. Jeroen Bosman (an excellent description of the issues): Science Mag sting of OA journals: is it about Open Access or about peer review?

17. John Hawks (interesting take): “Open access spam” and how journals sell scientific reputation

It is also nice to see some of them disclosing their particular set of conflicts in their discussions. I have none. And here are some more reactions on Twitter and a discussion hosted by Science:

5 thoughts on “Bohannon’s Science Sting – playing devil’s advocate and proposing a solution

  1. You wrote:
    “It took a journalist (albeit one with a PhD in molecular biology) to reveal the extent of the peer-review problems among predatory journals. ”
    I don’t think you draw the correct conclusions from this anecdote. Perhaps paraphrasing your sentence helps: “It took a journalist (albeit one with a PhD in molecular biology) to reveal that the Nigerian scam really is a scam.”

    I mean seriously now! You submit an article to a brand new journal that says “we’ll publish anything as long as you give us money!” and then the publish anything as long as you give them money. How earth shattering is that? You really need a PhD to figure that out?

    Moreover, each of these journals then apparently accepted *one* bogus article. If this means ‘peer-review problems’, then both Science and Nature (and essentially all other journals) have massive peer-review problems, as they also have at least one (and surely many more) bogus articles which are published and not retracted. Just check #arseniclife for Science or this gem for Nature:
    http://blogarchive.brembs.net/comment-n519.html

    Selection bias and missing control groups render this op/ed article in Science completely useless for any sort of analysis or conclusions. The ‘results’ presented there are completely and utterly meaningless. The fixes to this ‘sting’ are so easy and so obvious, that one can only question the motives of the author. If the fixes are not obvious to the author, he should not be in the business of science journalism, as he clearly lacks the necessary training.

    Perhaps another analogy to make it more clear: what the author did was analogous to taking 300 patients with the common cold, give them all homeopathic treatment and report “homeopathy works” when 62% of them get better a week after ‘treatment’. I hope it becomes obvious that, essentially anybody taking this ‘sting’ seriously, essentially disqualifies themselves from any evidence-based discussion on the matter.

    • I would say that the results give us some small hints that Beall’s list is a reasonably good estimate of the problematic journals, even though accepting one article does not prove or disprove the quality of any of the journals individually. When the percentage is so different between the two groups, it starts to look like a pattern. It’s also a good hypothesis-generating news article because it might inspire people to fix the problematic design of the ‘sting’ and try again. Why not implement the whole thing as a form of continual quality assurance, and monitor the DOAJ (and subscription publishers?) for problematic journals that have slipped through without being caught out?

      I tend not to think of the news article as a definitive study, and more like shining another torch into a (mostly) dark room. It’s then up to everyone else to find the switch to turn the light on and measure the problems properly. You have already done a lot of good work in the area, so perhaps the ‘sting’ (or its problems) will inspire more people like you to take a closer look? After all, aren’t we all working towards the same goals?

      • I think any single look at *many* journals on Beal’s list is probably about as telling as the emails with millions of dollars in foreign bank accounts just waiting for our rescue are. No uncontrolled anecdote required, let alone a smear of virtually all competitors of Science Magazine. The difference between Beal’s list and DOAJ is absolutely meaningless, because of the study design (e.g. selection bias).

        There have been a few attempts at such ‘sting’ operations before, see e.g.
        http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844#.Uk5_q5CK_Vs.twitter
        This can actually be done in ways that tell us something and it’s not all that difficult. That Bohannon failed to do the easy things is rather telling in a few different ways.

        If you really still think this news article shines some light on anything, you may be at risk of being susceptible to falling prey to homeopathic ‘studies’ as well. I suggest you read up on some basic experimental design / statistics. I’m afraid this op/ed piece (not even news!) is completely worthless, IMHO.

        • 1. No, it is more like publishing 304 case studies in a new strain of influenza virus for which there is a paucity of data. This form of evidence is common, and an important first step, typically used to generate hypotheses that can be tested in controlled studies.

          2. I’m sorry that by trying to find the positive aspects of a (flawed) study that could be used to improve the current state of open access annoyed you, but I stand by my positive perspective and I will not accuse Science of any sort of bias or conspiracy based on a single news article.

          3. I understand you have a different perspective about the news article but I don’t think there is a need for an ad hominem argument about a junior researcher’s abilities in experimental design and statistics. So let’s leave it there.

  2. ¿Quién teme a la revisión por pares? Demos un voto de confianza al sistema | Psicoteca

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s