So you’ve found a competing interest disclosure. Now what?

Published research varies across a spectrum that at one end is simply marketing masquerading as genuine inquiry. Actors in lab coats. To counter this problem, every time research is published in a journal, the authors are expected to declare anything that might have affected their impartiality in that work. Unfortunately, we very rarely do anything with those disclosures. It is as if by disclosing a potential competing interest, any effects on the research are supposed to either magically disappear or readers will somehow be able to magically account for their influence on the conclusions.

Let’s say you are reading a news story about a clinical trial that shows multivitamins will help your children grow up to be smarter, taller, healthier, and stronger. Seems to good to be true? You ask: “Is there a chance that the research has been distorted to make it look better than it really is?” so you try to find a link to the actual article so that you can check to see who the researchers are… and evaluate the quality of the research to determine its validity in context.

It’s actually much harder to do than it sounds, because for a substantial proportion of published articles, authors have failed to disclose things that would be defined as a potential competing interest by any standard definition. And in most cases, the competing interest disclosures are hidden behind paywalls, so you won’t be able to check the disclosures unless you have a subscription (or pay to “rent” the article, or use Sci-Hub to access it for free).

Then you ask: “What should I actually do if I encounter a competing interest disclosure?” At the moment, you have one of the following options: (a) you could either ignore the disclosure and take the research at face value; (b) you can throw the research out and ignore it because the research findings may be compromised; or (c) you could apply the best tools we have available for measuring the risk of bias even though we know they won’t always catch the nuanced ways in which research designs and reporting can be distorted.

Or more simply:  ¯\_(ツ)_/¯

So while we know that competing interests cause the kinds of biases that can lead to widespread harm, they also introduce substantial uncertainty into biomedical research because we simply don’t know if we can safely use research findings to inform our decision-making regardless of whether the authors have disclosed their potential competing interests or not.

But I think a solution to the problem is on its way. The first and most important step in that solution is to bring competing interests disclosures out into the open in ways that we can actually use. We need them to be made public, structured in a proper taxonomy instead of a bunch of free text descriptions, and accessible – in ways that both humans and machines can interpret and make use of.

That is why we think a comprehensive public registry of competing interest disclosures for all researchers is a good idea. People have been working on similar ideas for a while, and we have put many of these things in the review we published in the new journal, Research Integrity and Peer Review.

  • If you are interested, be sure to check out the IOM report (check the review), and read between the lines to try and understand why there might have been some disagreements about the best ways to deal with competing interests. This work eventually led to Convey, which has some similar goals but is not necessarily aimed at being comprehensive, and seems to be progressing nicely towards a different kind of goal from the updates on the webpage.
  • One of the things we didn’t include in the review because I hadn’t seen it until too late, is this browser plugin, which uses information from PMC to display funding and competing interests statements alongside the abstracts in PubMed. You could always click on the PMC link and scroll down to try and find them, but this is a neat idea.
  • It turns out that the idea for creating a standard list of competing interests and maintaining them in a public space was proposed as early as 2007, by Gordon Rubenfeld, in a letter to The Lancet. Maybe it is finally the right time to do this properly.
  • If you are a researcher and you like what you see in the first issue of the Research Integrity and Peer Review journal, then please consider publishing your research there. Besides the very interesting and high-profile editorial board, you might even have your manuscript handled by me as an associate editor.

Of course there are more things that will need to be done once we can manage for a much more comprehensive, consistent, and accessible way to disclose competing interests in research, but those come down to treating competing interests just like any other confounding variable. We are currently working on both methods and policies that will help us populate the registry in a longitudinal fashion (i.e. a timeline for every researcher who has published anything in biomedical research), and keep it up to date. We are also working on ways to take existing free text disclosures and classify them according to how much of an influence they have had over research findings in the past, and on a scale that has been virtually impossible until very recently.

Also, check out a more precise description of the idea in this related opinion piece published in Nature today. As usual, I will add to this post and link to any responses, media, comments, interesting tweets, etc. below as I spot them online.

Twitter users with anti-vaccine opinions are relatively easy to spot if we can measure their misinformation exposure

So…I have been systematically collecting tweets about human papillomavirus (HPV) vaccines since October 2013. We now have over two hundred thousand tweets that included keywords related to HPV vaccines, and the first of two pieces of research we have undertaken using these data has just been published in the Journal of Medical Internet Research. It covers 6 months, 83,551 tweets from 30,621 users connected to each other through 957,865 social connections. The study question is a relatively simple one – we wanted to find out about how many people are tweeting “anti-vaccine” opinions about HPV vaccines, the diversity of their concerns, and how the misinformation exposure is distributed throughout the Twitter communities.

What we found was in some ways surprising – around 24% of the tweets about HPV vaccines were classified as “negative” (more on this later). To me, this seems like a very large proportion given that only around 2% of adults are actually refusing vaccinations for their children. In other ways, I’m less surprised because of how many people have so many other unusual beliefs, and the number of surveys that suggest that 20% to 30% of adults believe that vaccines cause autism.

Looking at how people follow each other within the group of 30,621 users, we found that around 29% of everyone who tweeted about HPV vaccines were exposed to a majority of these “negative” tweets because of who they follow.

To classify the tweets as either “negative” or “neutral/positive”, we used supervised machine learning classifiers that were slightly different to the normal kinds of classifiers that just use information about the text to examine the sentiment of a tweet. I’ll be talking about these machine learning classifiers at the MEDINFO conference in Sao Paulo this August.

What we really wanted to know was how many Twitter users were being exposed to this negative kind of information – usually anecdotes about harm, conspiracy theories, complete fabrications, or some strange amalgamation of all of them – whether these users mostly grouped together, and how far their information reached across communities that might be making decisions about HPV vaccines for themselves or their children.

exposure_follower_network
A network of 30,621 Twitter users who posted tweets about HPV vaccines in a six month period. Users in orange were exposed mostly to negative opinions. Circles are users, larger ones have more followers within this group of users. Users more closely connected are generally positioned closer to each other in the picture.

We also wanted to know a bit more about the reach of the actual science and clinical evidence that is being published in the area. As researchers, we know that there are now studies showing that the HPV vaccine is safe and that there is early evidence of effectiveness in the prevention of cervical cancer, but we don’t really know who might be “exposed” to that kind of information.

Perhaps unsurprisingly, the people producing the science of HPV vaccines were located pretty much as far away as they could possibly be from the people exposed mostly to negative opinions. Most of the tweets linked directly to peer-reviewed articles came from the people in the very top left section of the network illustration above.

The main contribution of our study was to determine how much more likely it is that a user who was previously exposed to negative opinions would be to then tweet a negative opinion. The answer was: “a lot more”.

But to address the reasons why users’ opinions were relatively easy to predict if we know about the information they were exposed to in the past, we have to do a lot more work…

It could be that the opinions were “contagious” and spread through the community. It might also be that people end up forming “homophilous” connections with other users who express the same negative opinions about HPV vaccines. The much more likely explanation is that people who share opinions about all kinds of other things besides HPV vaccines (like guns, religion, politics, conspiracies, organic vegetables, crystals, and magical healing water) are more likely to be connected to each other, and their opinions about HPV vaccines are due to the breadth of misinformation that spreads to them from influential news organisations, celebrities, friends, and magical water practitioners.

It is important that we are careful to explain that the study only demonstrates an association between what people are exposed to in the past, and the direction of their expressed opinions after that. It does not show causation, and it does not tell us how those people came to believe what they do.

The study does tell us something important about how we might be able to estimate risks of poor vaccination decision-making within particular communities in space and time. One of the things we would like to be able to do is to examine where the concentrations of misinformation exposure are distributed geographically in a couple of countries (US and Australia – because that is where we know best), as a way of helping public health organisations better understand who might be vaccine anxious (or at risk of becoming vaccine anxious), and the specific concerns they might have. Because remember, only 2% of adults are conscientiously refusing to vaccinate their children, but an awful lot more of them might be forming their opinions based on the awful misinformation that spreads through the communities they inhabit.

How to predict the conclusion of a review without even reading it…

Short version: We published a new article in the Journal of Clinical Epidemiology all about selective citation in reviews of neuraminidase inhbitors – like Tamiflu and Relenza.

Lots of reviews get written about drugs (especially the ones that get prescribed often), and the drugs used to treat and prevent influenza are no exception. There are more reviews written than there are randomised controlled trials, and I think it is hard to justify why doctors and their patients would need so many different interpretations of the same evidence. When too many reviews are written in this way, I like to call it “flooding”.

The reason for why there are so many reviews written probably has something to do with a problem that has been written about many times over by people much more eloquent than I am: when marketing is disguised as clinical evidence.

We recently undertook some research to try and understand how authors of reviews (narrative and systematic) manage to come up with conclusions that appear to be diametrically opposed. For the neuraminidase inhibitors (e.g. Tamiflu, Relenza), conclusions in reviews varied from recommending the early use in anyone who looks unwell, or massive global stockpiling for preventative use, to others who question the very use of the drug in clinical practice and raise safety concerns. We hypothesised that one of the ways these differences could manifest in reviews was through something called selective citation bias.

Selective citation bias happens when authors of reviews are allowed to pick and choose what they cite in order to present the evidence in ways that fit their predetermined views. And of course, we often associate this problem with conflicts of interest. This has in the past led to drugs being presented as safe and effective (repeatedly) when they simply aren’t.

By the way, here’s a picture of approximately where I am right now while I’m writing this quick update. I’m on a train between Boston and NYC in the United States, passing through a place called New Haven.

train

To test our hypothesis about selective citation bias, we did something quite new and unusual using the citation patterns among the reviews of neuraminidase inhibitors. We looked at 152 different reviews published since 2005, as well as the 10,086 citations in the reference lists pointing at 4,574 unique articles. Two members of the team (Diana and Joel) graded the reviews as favourable or otherwise, and when they both agreed that the review presented the evidence favourably, we put that in the favourable pile. The majority of reviews (61%) ended up in this group.

We then did two things: we undertook a statistical analysis to see if we could find articles that were by themselves much more likely to be cited by favourable reviews. And we constructed a bunch of classifiers using supervised machine learning algorithms to see how well we could predict which reviews were favourable by looking only at the reference lists of the articles.

What we found was relatively surprising – we could predict a favourable conclusion with an accuracy of 96.5% (in sample) only using the reference lists, and without actually looking at the text of the review at all.

A further examination of the articles that were most useful (in combination) for predicting the conclusions of the reviews suggested that the not-favourable pile tended to cite studies about viral resistance much more often than their favourable counterparts.

What we expected to find, but didn’t, was that industry-funded studies would be over-represented in favourable reviews. To me, the lack of a finding here means that the method we devised was probably better at finding what was “missing” from the reference lists of the majority rather than what is over-represented in the majority. The maths on this makes sense too.

So we think that applying machine learning to the metadata from published reviews could be useful for editors trying to review new narrative reviews. More importantly, when faced with multiple reviews that clearly disagree with each other, these methods could be used to help identify what’s missing from reviews in order to restore some balance in the representation of primary clinical evidence in things like reviews and guidelines.

Media collection about conflicts of interest in systematic reviews of neuraminidase inhibitors

As usual, I’m keeping a record of major stories in the media related to a recently published paper. I will continue to update this post to reflect the media response to our article in the Annals of Internal Medicine.

When I checked last (1 May 2015), the Altmetric score was 112. Here’s the low-tech way to check that…

Capture2

Should we ignore industry-funded research in clinical medicine?

A quick update to explain our most recent editorial [pdf] on evidence-based medicine published in the Journal of Comparative Effectiveness Research. It’s free for anyone to access.

What do we know?

Industry funded research does not always lead to biases that are detrimental to the quality of clinical evidence, and biased research may come about for many reasons other than financial conflicts of interest. But there are clear and strong systemic differences in the research produced by people who have strong financial ties to pharmaceutical companies and other groups. These differences have in the past been connected to problems like massive opportunity costs (ineffective drugs) and widespread harm (unsafe drugs).

[Spoiler alert]

What do we think?

Our simple answer is no, we don’t think that industry-funded research should be excluded from comparative effectiveness research. To put it very simply, around half of the completed trials undertaken each year are funded by the industry, and despite the overwhelming number of published trials we see, we still don’t have anywhere near enough of them to properly answer all the questions that doctors and patients have when trying to make decisions together.

Instead, we think improvements in transparency and access to patient-level data, the surveillance of risks of bias, and new methods for combining evidence and data from all available sources at once are much better alternatives. You can read more about all of these in the editorial.

Bonus:

Also, check out the new article from our group on automated citation snowballing published in the Journal of Medical Internet Research. It forms the basis of a recursive search and retrieval method that finds peer-reviewed articles online, downloads them, extracts the reference lists, and follows those links to find and retrieve articles recursively. It is particularly interesting because it can automatically construct citation networks back from a single paper.

Guerilla open access, public engagement with research, and ivory towers

Despite the growth of open access publishing, there is still a massive and growing archive of peer-reviewed research that is hidden behind paywalls. While academics can reach most of the research they need through library subscriptions, researchers, professionals and the broader community outside of academia are effectively cut off from the vast majority of peer-reviewed research. If the growth of file sharing communities transformed the entertainment industry more than fifteen years ago, is a similar transformation in academic publishing inevitable?

Together with Enrico Coiera and Ken Mandl, I published an article today in the Journal of Medical Internet Research. In the article, we considered the plausibility and consequences of a massive data breach and leak of journal articles onto peer-to-peer networks, and the creation of a functioning decentralised network of peer-reviewed research. Considering a hypothetical Biblioleaks scenario, we speculated on the technical feasibility and the motivations that underpin civil disobedience in academic publishing.

It appears as though academics are not providing pre-print versions of their article anywhere near as often as they could. For every 10 articles published, 2 or 3 can be found online for free, but up to 8 of them could be uploaded by the authors legally (this is called self-archiving, where authors upload pre-print versions of their manuscripts). Civil disobedience in relation to sharing articles is still quite rare. Examples of article-sharing on Twitter and via torrents have emerged in the last few years but only a handful of people are involved. There it not yet a critical mass of censorship-resistant sharing that would signal a shift into an era of near-universal access like we saw in the entertainment industry in the late 1990s.

However, as the public come to expect free access to all research as the norm rather than the exception, it might be more likely that the creation of an article-sharing underground will come from outside academia. What is unknown is whether or not the public actually want to access peer-reviewed research directly. From the little evidence that is available on this question, it seems that doctors, patients, professionals of all kinds, as well as the broader community might all benefit from the creation of an underground network of article-sharing, and it may even serve to reduce the gap between research consensus and public opinion for issues like climate change and vaccination, where large sections of the broader community disagree with the overwhelming majority of scientific experts.

Given the size of recent hacks on major companies, there appears to be no technical barriers to a massive data breach and leak. However, by removing the motivations behind a Biblioleaks scenario, publishers and researchers might be able to avoid (or skip over) a period of illegal file-sharing. University librarians could build the servers that would seed the torrents for pre-prints, helping to ensure quality control and improving the impact of the research in the wider community. Researchers can and should learn the self-archiving policies for all their work and upload their manuscripts as soon as they are entitled or obliged to do so. Prescient publishers might find ways to freely release older articles on their own websites to avoid losing traffic and advertising revenue.