Europe's medicines regulator, criticised in the past for excessive secrecy, is opening its data vaults to systematic scrutiny in a move that will let independent researchers trawl through millions of pages of clinical trial information.

The change is a landmark in transparency that puts Europe ahead of the United States, according to critics of the $1 trillion-a-year global drugs industry, who have long argued for full access to trial data.

Such information is a treasure trove for scientists wanting to test drug company claims and potentially expose product deficiencies.

As part of a process of opening up, the European Medicines Agency (EMA) plans to hold a conference in November to consider ways of making large data sets available rapidly and routinely to outside investigators.

The shift chimes with a widespread push in many fields to treat scientific data as a public rather than a private resource as the world grapples with a growing flood of information.

It is blow for the pharmaceutical industry, which guards its commercial secrets fiercely and has not before been required to share its data with independent researchers or academics. Companies however have little choice in the matter since they must submit their data to the regulator to get drugs approved.

"It's a sea-change in attitude," said EMA senior medical officer Hans-Georg Eichler, who admits regulators and drug companies have been tarnished by past scandals.

The EMA, like the Food and Drug Administration (FDA) in the United States, was criticised for failing to spot problems with drugs such as Merck & Co's now withdrawn painkiller Vioxx and GlaxoSmithKline's diabetes pill Avandia.

Fraud settlement

Access to data was important in both cases. GSK's failure to give the FDA safety information about Avandia was one factor behind its record $3 billion healthcare fraud settlement in the United States on July 2.

"This industry has certainly done a disservice to itself," Eichler said in an interview at the agency's headquarters in east London.

"I hope everybody will learn that daylight is the best disinfectant and this will be a contributing step in rebuilding trust in the regulator and in the industry."

The EMA's change of heart has not been entirely voluntary.

Its position used to be that data from clinical trials paid for by industry was commercially confidential, but it was jolted from that stance after the European Ombudsman ruled such confidentiality was not compatible with the public interest.

There has also been a cultural shift at the agency and transparency has been put at the top of the agenda by the EMA's new executive director, Guido Rasi. Rasi took over in January from Thomas Lonngren, who controversially became a consultant for the pharmaceutical industry within weeks of leaving.

Peter Gotzsche, director of the nonprofit Nordic Cochrane Centre in Denmark, which carries out systematic healthcare reviews, is one of those who battled the EMA and successfully appealed to the ombudsman to get access to data. He is convinced the about-turn will make a major difference.

"It's very good news. Europe used to be behind the FDA in terms of openness and transparency but now Europe is ahead," he said.

"It will make a huge difference to public health. The pharmaceutical industry routinely exaggerates the benefits of its drugs and plays down the harms or hides them. By getting access to material held by drug regulators we may discover that many drugs are less effective than we thought they were and also more harmful."

In the last 18 months, the EMA has released around 1.5 million pages of clinical trial data - an increase of more than a hundred-fold compared to 2010 and 2009.

Raw data

But providing that information at the moment is laborious and time-consuming, since patient-level data must be redacted by staff to protect patient confidentiality.

In future, Eichler hopes standardised systems can be agreed for anonymising data, although the devil will be in the detail of setting the parameters for this procedure.

In addition to uncovering new facts about drugs through secondary analysis, giving researchers access to raw data might one day open up new methods for treatment - perhaps using machine learning systems that could marry a patient's health record directly to data from appropriate clinical trials.

Getting access to large patient-level data sets will be central to developing the computer science and artificial intelligence systems needed for such tools.

Europe's drugs watchdog, however, does not envisage a complete free-for-all for data, which would only be released after the agency has finished reviewing a new drug.

"We don't want to be blind-sided by studies of poor quality that create a public health scare," Eichler said. "What the debate now is about is finding the right conditions to minimise the potential for false findings."

One way to achieve that would be to make researchers spell out upfront how data will be used by establishing a clear protocol that cannot be revised after the event.

That idea is backed by GSK, which has gone further than some companies in providing access to data following past rows.

A spokesman for Britain's biggest drugmaker said it already provided patient-level data to external researchers, on a case by case basis, on condition protocols were peer reviewed ahead of time and there was a commitment to publish results.