Guerilla open access, public engagement with research, and ivory towers

Despite the growth of open access publishing, there is still a massive and growing archive of peer-reviewed research that is hidden behind paywalls. While academics can reach most of the research they need through library subscriptions, researchers, professionals and the broader community outside of academia are effectively cut off from the vast majority of peer-reviewed research. If the growth of file sharing communities transformed the entertainment industry more than fifteen years ago, is a similar transformation in academic publishing inevitable?

Together with Enrico Coiera and Ken Mandl, I published an article today in the Journal of Medical Internet Research. In the article, we considered the plausibility and consequences of a massive data breach and leak of journal articles onto peer-to-peer networks, and the creation of a functioning decentralised network of peer-reviewed research. Considering a hypothetical Biblioleaks scenario, we speculated on the technical feasibility and the motivations that underpin civil disobedience in academic publishing.

It appears as though academics are not providing pre-print versions of their article anywhere near as often as they could. For every 10 articles published, 2 or 3 can be found online for free, but up to 8 of them could be uploaded by the authors legally (this is called self-archiving, where authors upload pre-print versions of their manuscripts). Civil disobedience in relation to sharing articles is still quite rare. Examples of article-sharing on Twitter and via torrents have emerged in the last few years but only a handful of people are involved. There it not yet a critical mass of censorship-resistant sharing that would signal a shift into an era of near-universal access like we saw in the entertainment industry in the late 1990s.

However, as the public come to expect free access to all research as the norm rather than the exception, it might be more likely that the creation of an article-sharing underground will come from outside academia. What is unknown is whether or not the public actually want to access peer-reviewed research directly. From the little evidence that is available on this question, it seems that doctors, patients, professionals of all kinds, as well as the broader community might all benefit from the creation of an underground network of article-sharing, and it may even serve to reduce the gap between research consensus and public opinion for issues like climate change and vaccination, where large sections of the broader community disagree with the overwhelming majority of scientific experts.

Given the size of recent hacks on major companies, there appears to be no technical barriers to a massive data breach and leak. However, by removing the motivations behind a Biblioleaks scenario, publishers and researchers might be able to avoid (or skip over) a period of illegal file-sharing. University librarians could build the servers that would seed the torrents for pre-prints, helping to ensure quality control and improving the impact of the research in the wider community. Researchers can and should learn the self-archiving policies for all their work and upload their manuscripts as soon as they are entitled or obliged to do so. Prescient publishers might find ways to freely release older articles on their own websites to avoid losing traffic and advertising revenue.

Neuropsych trials involving kids are designed differently when funded by the companies that make the drugs

Over the short break that divided 2013 and 2014, we had a new study published looking at the designs of neuropsychiatric clinical trials that involve children. Because we study trial registrations and not publications, many of the trials that are included in the study are yet to be published, and it is likely that quite a few will never be published.

Neuropsychiatric conditions are a big deal for children and make up a substantial proportion of the burden of disease. In the last decade or so, more and more drugs are being prescribed to children to treat ADHD, depression, autism spectrum disorders, seizure disorders, and a few others. The major problem we face in this area right now is the lack of evidence to help guide the decisions that doctors make with their patients and their patients’ families. Should kids be taking Drug A? Why not Drug B? Maybe a behavioural intervention? A combination of these?

I have already published a few things about how industry and non-industry funded clinical trials are different. To look at how clinical trials differ based on who funds them, we often use the registry, which currently provides information for about 158K registered trials and is made up of about half US trials, and half trials that are conducted entirely outside the US.

Some differences are generally expected (by cynical people like me) because of the different reasons why industry and non-industry groups decide to do a trial in the first place. We expect that industry trials are more likely to look at their own drugs, the trials are likely to be shorter, more focused on the direct outcomes related to what the drug claims to do (e.g. lower cholesterol rather than reduce cardiovascular risk), and of course they are likely to be designed to nearly always produce a favourable result for the drug in question.

For non-industry groups, there is a kind of hope that clinical trials funded by the public will be for the public good – to fill in the gaps by doing comparative effectiveness studies (where drugs are tested against each other, rather than against a placebo or in a single group) whenever they are appropriate, to focus on the real health outcomes of the populations, and to be capable of identifying risk-to-benefit ratios for drugs that have had questions raised about safety.

The effects of industry sponsorship on clinical trial designs for neuropsychiatric drugs in children

So those differences you might expect to see between industry and non-industry are not quite what we found in our study. For clinical trials that involve children and test drugs used for neuropsychiatric conditions, there really isn’t that much difference between what the industry choose to study and what everyone else does. So even though we did find that industry is less likely to undertake comparative effectiveness trials for these conditions, and the different groups tend to study completely different drugs, the striking result is just how little comparative effectiveness research is being done by both groups.

journal.pone.0084951.g003 (1)

A network view of the drug trials undertaken for ADHD by industry (black) and non-industry (blue) groups – each drug is a node in the network; lines between them are the direct comparisons from trials with active comparators.

To make a long story short, it doesn’t look like either side are doing a very good job of systematically addressing the questions that doctors and their patients really need answered in this area.

Some of the reasons for this probably include the way research is funded (small trials might be easier to fund and undertake), the difficulties associated with acquiring ethics and recruiting children to be involved in clinical trials, and the complexities of testing behavioural therapies and other non-drug interventions against and with drugs.

Of course, there are other good reasons for undertaking trials that involve a single group or only test against a placebo (including safety and ethical reasons)… but for conditions like seizure disorders, where there are already approved standard therapies that are known to be safe, it is quite a shock to see that nearly all of the clinical trials undertaken for seizure disorders in children are placebo-controlled or are tested only in a single group.

What should be done?

To really improve the way we produce and then synthesise evidence for children, we really need to consider much more cooperation and smarter trial designs that will actually fill the gaps in knowledge and help doctors make good decisions. It’s true that it is very hard to fund and successfully undertake a big coordinated trial even when it doesn’t involve children, but the mess of clinical trials that are being undertaken today often seem to be for other purposes – to get a drug approved, to expand a market, to fill a clinician-scientist’s CV – or are constrained to the point where the design is too limited to be really useful. And these problems flow directly into synthesis (systematic reviews and guidelines) because you simply can’t review evidence that doesn’t exist.

I expect that long-term clinical trials that take advantage of electronic medical records, retrospective trials, and observational studies involving heterogeneous sets of case studies will come back to prominence for as long as the evidence produced by current clinical trials is hampered by compromised design, resource constraints, and a lack of coordinated cooperation. We really do need better ways to know which questions need to be answered first, and to find better ways to coordinate research among researchers (and patient registries). Wouldn’t it be nice if we knew exactly which clinical trials are most needed right now, and we could organise ourselves into large-enough groups to avoid redundant and useless trials that will never be able to improve clinical decision-making?

Bohannon’s Science Sting – playing devil’s advocate and proposing a solution

[Update: I realise that perhaps many of you are not going to have the same perspective about the Science Sting that I have purposefully taken here (hence the title). Apologies in advance.]

It took a journalist (albeit one with a PhD in molecular biology) to reveal the extent of the peer-review problems among predatory journals. Bohannon submitted fatally flawed and boringly bad articles to a set of open access journals that charge fees to publish. Of 255 submissions with results, 157 journals (62%) accepted the article. This did not come as a surprise to a lot of people who have been watching or interacting with this vast underworld of predatory publishing.

Simply, fee-charging predatory open access publishers make more money if they accept more articles. It’s not going to be good for business in the long-term (because of reputation problems) but it seems to be working quite well for a number of publishers right now. The need/desire for profits is also a big problem in subscription journals but often for very different reasons (where profit relies on denying access).

But is it working for them? There is limited evidence to suggest that business is good for predatory journals. Hindawi (previously listed as borderline predatory) apparently makes 52% profits, which is phenomenal compared to the already obscene 36% operating profits reported by Elsevier. The reason why business is so good is the direct consequence of the publish or perish mentality that pervades academia. Add to that the hundreds of thousands of students graduating with PhDs each year, the low overheads associated with starting a journal, and paid gold open access becomes an attractive business proposition for any shonky operator.

Response from those heavily invested in open access

Bohannon tells us that he talked about the work with “a small group of scientists who care deeply about open access”, and in the news article explained carefully that the growth of open access has multiplied the problem of predatory journals. This is because predatory publishers are much more likely to opt for a paid gold open access model. I don’t think anyone can disagree with that. The problem is that people who care deeply about open access appear to have often interpreted the news article in a defensive way because of their particular perspectives on open access. There’s a whole bunch of responses to the article that come from the full spectrum of open access advocates and I have made a list below.

To play the devil’s advocate, here’s what they have often missed in their responses.

  • Bohannon correctly included the suggestion that targeting the low end of subscription journals could produce the same result and explicitly indicated that he did not examine them.
  • The article is directed at a particular subset of journals that charge fees and are listed on the DOAJ or on Beall’s list. The aims match the selection of journals and both are clear. At no point does Bohannon say that the chosen set represents open access generally.
  • It is an article about peer review. The fact that the targeted journals were all paid gold open access journals is important to note but that point has very little to do with the huge problems in peer review, which we all know are pervasive and a bigger problem in gold open access.

The results also show that Beall’s list does a reasonably good job of compiling the problems. It was nice to see that Beall was (mostly) vindicated in his identification of dodgy publishers. Over 80% of the journals on Beall’s list accepted the article after some kind of a review. For journals on DOAJ, the proportion was 45%. Much lower but still substantial.

It was unsurprising that Beall’s list produced the highest proportion of accepted articles and the highest proportion of missing peer review, that DOAJ produced fewer in each of the two. Glass-half-full types would have focused more heavily on this to show that there were in fact plenty of good open access journals that do charge fees, do undertake peer review or reject on first principles, and noted that even among a targeted group of journals accused of predatory behaviour, some still undertook peer review.

So what?

There are two important things we need to remember when interpreting the results of the Bohannon’s Science Sting. Firstly, predatory journals and open access journals are not synonymous but there are a lot of predatory journals that are open access. Bohannon did not conflate them in his report and explained his methods better than most journal articles I’ve seen recently. Secondly, if open access is to eventually replace the subscription model completely, then the people that are best placed to tackle predatory journals using open access models are the people that are already measuring, curating, listing and analysing open access. Houses in order, so to speak.

DOAJ should be the first to act – directly with the journals that accepted the paper, and then with the wider group of fee-charging journals. It doesn’t matter what proportion of subscription articles *would* have accepted the papers, the problem still exists no matter how much the “lack of a proper control” or “Science conspiracy against open access” arguments get thrown around to dilute the message.

Solving the problem

If you want a radical solution to the problem of bad peer review in predatory journals, there’s an obvious one that no one seems to be suggesting:

Remove profits from publishing entirely.

The method is simple. Only index and recognise articles that are published with non-profit and flat-rate subsidised (platinum open access) journals – like some of those published by the various scientific societies. Don’t pay journals for “how much” but for “how well”. If subscription journals and for-profit open access journals can’t be cited, indexed, or contribute to career progression, then the market for predatory journals disappears.

What others have been saying

Here is a list of other responses to Bohannon’s Science Sting.

1. Graham Steel: The Publishing “Sting”, the reaction, and the outcome

2. Åse Innes-Ker: A publishing sting, but what was stung?

3. Björn Brembs: Science Magazine Rejects Data, Publishes Anecdote

4. Claire Shaw: Hundreds of open access journals accept fake science paper

5. Peter Suber: New “sting” of weak open-access journals.

6. Curt Rice: What Science — and the Gonzo Scientist — got wrong: open access will make research better

7. Martin Eve: Flawed sting operation singles out open access journals  – and a longer version

8. Ernesto Priego: Who’s Afraid of Open Access?

9. Nigel Hawkes: Spoof research paper is accepted by 157 journals

10. Michael Eisen: I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals

11. Mike Taylor (SV-POW): John Bohannon’s peer-review sting against Science

12. Fabiana Kubke (makes good points about navigating open access): Science gone bad

13. Lenny Teytelman (interesting take & new analysis): What hurts science – rejection of good or acceptance of bad?

14. The Directory of Open Access Journals: DOAJ’s response to the recent article in Science entitled “Who’s Afraid of Peer Review?”

15. Open Access Scholarly Publishers Association: OASPA’s response to the recent article in Science entitled “Who’s Afraid of Peer Review?”

16. Jeroen Bosman (an excellent description of the issues): Science Mag sting of OA journals: is it about Open Access or about peer review?

17. John Hawks (interesting take): “Open access spam” and how journals sell scientific reputation

It is also nice to see some of them disclosing their particular set of conflicts in their discussions. I have none. And here are some more reactions on Twitter and a discussion hosted by Science:

Do people outside of universities want to read peer-reviewed journal articles?

I asked a question on Twitter about whether or not people actually tried to read the peer-reviewed journal articles (not just the media releases), and if they encountered paywalls when they tried. This is what happened:

[Click on the time/date to see the conversation]

In case you don’t want to read through the whole conversation, it turns out that every person who answered the question said that they have in the past tried to access peer-reviewed journal articles, and that they have been stopped by paywalls. Some said it happened all the time.

There is very little evidence to show the prevalence of access and blocked access by the “interested public” for peer-reviewed journal articles. Some people seem to assume that only other scientists (or whatever) would be interested in their work, or that everything the “public” need to know is contained in a media release or abstract.

I think the results tell us a lot about the consumption of information by the wider community, the importance of scientific communication, the problem with the myth that only scientists want to read scientific articles, and the great need for free and universal access to all published research.

So far, I’ve been collecting whatever evidence I can get my hands on to relate to this question, especially in medicine, and I’ll add these pieces one by one below, just in case you are interested.

  1. Open access articles are downloaded and viewed more often than other articles, even when they do not confer a citation advantage. This is seen as evidence that people not participating in publishing are accessing the information.Davis, P.M., Open access, readership, citations: a randomized controlled trial of scientific journal publishing. The FASEB Journal, 2011. 25(7): p. 2129-2134.
  2. A Pew Internet Report found that one in four people hit a paywall when searching for health information online. Perhaps more importantly, that 58% of all people have looked for health information online (and in a country where only 81% use the Internet).
  3. From a UNESCO report on the development and promotion of open access: “First, it is known that [people outside of academia] use the literature where it is openly available to them. For example, the usage data for PubMed Central (the NIH’s large collection of biomedical literature) show that of the 420,000 unique users per day of the 2 million items in that database, 25% are from universities, 17% from companies, 40% from ‘citizens’ and the rest from ‘Government and others’.”Swan A. Policy guidelines for the development and promotion of open access, United Nations Educational, Scientific and Cultural Organization, 2012, Paris, France. (Page 30). Available at: course, people accessing PubMed Central from domestic IP addresses might often be academics working late at night at home without a VPN (like I am doing now).

About fifty people responded to my question on Twitter. I realise that my audience is probably biased towards the highly-educated, informed, younger, and information-savvy, but I think there are clear and obvious groups of people outside of universities who would be interested in reading published research. These people include doctors, engineers and developers, parents of sick children, politicians and policy-makers, practitioners across a range of disciplines, museum curators, artists, and basically everyone with an interest in the world around them. That this aspect of open access hasn’t been the feature of many surveys or studies seems bizarre to me.

Perhaps most importantly, I think we need to know a lot more about just how often people outside of academia want to access published research, and if problems with access are stopping them from doing so.

Surely the impetus to move towards universal and open access to published research would grow if more academics realised that actually *everyone* wants access to the complicated equations, to the raw data and numbers, and to the authors’ own words about the breadth and limits of the research that they have undertaken.

All that glitters is not gold: the fallacy of open access evangelism

All-or-nothing open access evangelism perpetuates the problems of scientific publishing. Writing in the Guardian, one advocate has even suggested that publishing behind a pay-wall is immoral. That form of evangelism is wrong – for now – and may do more harm than good.

Yes, there are clear advantages to gold open access. Chief among these is that everyone gets unfettered access to research. There is no doubt that access is fundamental to the way science works. Yet there is a trend towards a simplistic kind of open access evangelism that seems to be gaining traction in mainstream academia, and it has me worried.

Problem 1. It is keeping costs of doing research unsustainable

In its current form, paid gold open access only shifts the cost of access from the libraries to the researchers without making it affordable. Where researchers and libraries are both publicly funded, money flows from the same founts to the same drains, just through different pipes. For hybrid policies (subscription-based journals with paid open access options), the public can sometimes pay twice.

Traditional publishers appear comfortable with gold and hybrid open access. Elsevier manages approximately two thousand journals and nearly 80% have open access options, costing between USD$500 and USD$5000 per article. The average cost of publishing an article in open access is a little over USD$900, and for biomedical disciplines the costs are far higher than that. There are low- or no-cost gold open access alternatives but they are not as popular and do not confer the same levels of prestige.

If advocates (and working groups in the UK) continue to promote the idea that gold open access is the appropriate direction for right now, publishing will remain too expensive for researchers with already restricted budgets, and the public will continue to fund ridiculously profitable publishing groups and shonky operators.

The obvious short-term solution, often proposed from within the open access community, is to focus on green open access (authors may freely distribute a version of the article), and dramatically improve the rate of self-archiving. Per article, library subscriptions may still be more expensive than open access fees, but a critical mass of self-archiving is a necessary step in the process.

The gap between the number of pay-walled articles that could be made available and the number that actually are, is a shameful indictment of academia. In a soon-to-be-formally-published article, Björk et al. have showed that 81% of published articles could have been self-archived, yet the uptake is around 12%. The blame for the gap is aimed directly at academics.

The civil disobedience of #icanhazpdf (Twitter users requesting and receiving articles from academics with institutional access) is nearly always against institutional policy. No one would recommend the practice, even if tools like TOR and anonymous public dropbox accounts could safely preserve anonymity. But it does provide a hint for what could be done with the 50 million articles already out there. Just as the legitimate use of Spotify replaced the illegal use of Napster for sharing music, we may soon see a tool that will be viewed as both the saviour and destroyer of academic publishing.

Problem 2. It permits the widespread ruination of quality and rigour in research

It is not just the largest publishing houses that are embracing pay-to-publish open access models. There appears to be an endless supply of researchers willing to engage with nearly 9000 open access journals. The number of journals has doubled since 2009, with more than three new journals established every day. While there are paragons of quality among the top tiers of open access journals, many have instead duped researchers and watered down the global research endeavour with misconduct, plagiarism, and inadequate peer-review.

Hindawi, a borderline predatory publisher, is reported to have a higher profit margin (52%) than Elsevier (36%), suggesting that publishing offers exceptional returns on investment, better than mining, pharmaceuticals, perhaps only beaten by illegal drug/human trafficking. Only a small fraction of open access journals have impact factors. Yes, there are gems among the bulk of gold open access journals (I’ve seen some), but many of these journals and the crap published in them only function to fill the research arena with work that adds very little to science, to innovation, and to improving society.

As a consequence of the publish-or-perish mentality and ease of publishing, scientific research has jumped way over the line from a curated library (little redundant information and sustained relevance) to the kind of vast streams of quickly-forgotten information that gives meaning to the phrase “sipping from the fire hose”.

This is where gold open access and creative commons (with attribution) licenses will eventually become vital. If academic publishing does reach a state in which access to all research is free, immediate and unrestricted, it will likely signal the biggest shift in science since the digital revolution. Much of the new research synthesis ideas involving algorithmic filtering, reanalysis, and meta-analysis are yet to be developed. Where I work, some of us have started to wander around the edges of these possibilities.

We may even be able to do away with journals completely. Michael Eisen, co-founder of PLoS, has argued that publishers add very little to the work done by researchers. Mathematicians have moved to take publishers out of the equation with the Episciences Project. In the future, scientists might open up their lab books and hard disks so that data and models can be freely shared, searched, recycled and linked together like vast open source software communities have been doing for years.

To get there, science needs better ways to attribute and praise individuals for discrete chunks of research. This is where altmetrics are expected to extend citation-based metrics to detail the full range of impact that research (not just publications) can have on scholarship and society. Using citations and journal impact factors to find good science is like trying to fish with explosives.

Lessons for researchers and research policy developers

While we wait for the future of research dissemination to emerge, there are simple ways in which academics can make sure they act in the best interests of the scientific community and the public.

Besides respecting alternative measurements of impact, funders should continue to mandate self-archiving through institutional repositories. Informaticians should investigate tools and motivations for sharing in line with the decentralized #icanhazapdf, the NIH manuscript submission system, Figshare, the Synaptic Leap, or the loophole that ResearchGate uses to encourage uploads to ‘personal’ pages.

Australians are relatively lucky when it comes to self-archiving. We have a mandate from National Health and Medical Research Council, and the Australian Research Council, that all publicly-funded research must be shared, and we have seen experts in open access discussing the cost-effectiveness of gold and green open access early and clearly.

As a bare minimum, academics must make their contact details public to entertain requests for inaccessible articles, check journal policies on open access prior to submission, avoid the temptation of predatory open access journals, and most importantly:

dramatically improve the woeful record in self-archiving.

There is no direct route to an academic publishing future where publicly-funded research outputs are both libre and gratis. A sustainable trajectory requires a diversity of affordable ways to disseminate research widely, but it will only work if we can retain our grip on the processes that ensure the rigour and quality being eroded.

[Image credit: The ENIAC]

How to do work-life balance: learn to say “no”

I wrote a piece for the Guardian’s Higher Education Network, all about the power of “no“. The piece was designed particularly with early-career researchers in mind, but there might be some resonance for researchers at other stages of their careers, and maybe even more widely.

I always struggle with turning down requests, which tends to make for an interesting, diverse, and very tiring career. I have absolutely enjoyed getting involved in a range of unusual and interesting research (and other) projects in the last six years but it has come at the cost of balance in my life. I’m pretty sure that I will continue to struggle with balancing work and whatever else it is that people are supposed to do when they are not working. At least writing about saying “no” has made me think about my own internal mechanisms for saying no.

In case you missed the link to the article I wrote, here it is: “Early career research: the power of ‘no’