New guide can help the public ask the right questions about data science claims

Guide aims to spark conversation between the public, policymakers and journalists about the quality and reliability of data and how it’s used

Data science guide
Data Science: A Guide for Society was created by Sense About Science, with Elsevier as a key supporter and collaborator. You can download it at the end of this story.

Back in November, I co-hosted an unusual workshop at the International Network of Government Science Advisors (INGSA) meeting in Tokyo. Sense About Science – an independent charity Elsevier partners with to promote awareness of research quality and peer review among the public, journalists and policymakers – wanted to test a framework for a public guide to data science.

Tracey Brown, OBE, Director, Sense about Science (center) and Errin Riley discuss the guide with Tim Farron MP at Evidence Week in the UK Parliament.With daily headlines touting the promise of big data and AI in predicting everything from the path of the next epidemic to end-of-season league positions of football teams to who is most likely to commit a crime, Sense about Science was asking what people needed to know in order to evaluate these claims.

The organization’s mission is to promote and defend the public interest in sound science and evidence. For Elsevier, as one of the largest providers of curated scientific content, data and information analytics, the project seemed a natural fit.

The project stemmed from a roundtable at the Broad Institute in Cambridge, Massachusetts. At Sense about Science’s request, Elsevier colleagues there hosted a small meeting with thinkers and statisticians from leading US institutions about the kinds of insights the public needed on data science. INGSA in Tokyo provided an opportunity to try them out.

I got the point. My own background is in engineering physics. When working at the Swedish Embassy in Tokyo as Science Counsellor, I was involved in our embassy’s reporting on the March 2011 Triple Disaster and notably the Fukushima Nuclear Accident. Official information was sparse and at times not well communicated – for example, radiation levels were given without any baseline on whether they were actually dangerous. That overall experience was an eye-opener as to how a guide to data science could support a better dialogue between decision makers, researchers and the public.

It was a huge task – to create a simple opening into an emerging science that could be relevant for people reading claims about Facebook data in the news as much as for government procurement officials buying in data services for health. Our Tokyo workshop, chaired by Prof. Yuko Harayama, proved that this was an important responsibility and that the challenge is global. It also showed that steering people through questions of quality and reliability in data science analysis was a daunting journey. But, eight months later, they are ready to publish.

I caught up with the publication’s coordinator, Errin Riley, and Sense about Science Director Tracey Brown on the eve of its publication.

The Rt Hon. the Lord Wallace of Tankerness QC discusses the guide with Errin Riley, Research and Policy Coordinator for Sense About Science, at Evidence Week in the UK Parliament.

When I met you, Tracey, you were frustrated about the lack of resources available to help people get involved in the conversations happening around data science.

TB: I was. I was particularly frustrated about discussions in Europe, where leading figures in policy and science were speaking at conference after conference despairing about the lack of respect for evidence and the rise of a ‘post-truth’ public. At the same time, they were announcing millions of euros on 'big data strategies' and hadn’t given thought or energy to asking whether people are equipped to ask the fundamental questions about quality. Here was an opportunity to equip citizens to be critically engaged in the evidence and decisions shaping their lives, and the people who could be leading that were too preoccupied to see it.

You were already taking matters into your own hands when we met in Tokyo. Tell me about what has happened since then. Errin, you’ve had the task of marshalling data scientists to the cause?

The Rt. Hon Stephen Timms MP discusses the guide with Dr. Hamid Khan, Public Engagement Coordinator for Sense about Science, at Evidence Week in the UK Parliament.ER: We have had some excellent input from researchers in different fields of data science. It was a bit of a battle for simplicity. And often a battle between different experts about precise definitions or the need to add lots of detail to simple statements. This rather missed the point.

How did you overcome that?

TB: In our discussions with data scientists, we challenged them to cut to the heart of an issue and drop as much technical language as possible. This was helped by getting across the idea to data scientists that we weren’t trying to write a data science textbook but to equip people with the language and basic concepts that would empower them to start a conversation about what weight different data analyses could bear.

ER: It has been hugely helpful for us to discuss it with people like Dr. Brad Fenwick  (Senior VP for Global Strategic Alliances at Elsevier), Dr. Michiel Kolman (Senior VP for Global Academic Relations at Elsevier) and some of the team who work with Prof. Sir Peter Gluckman, chair of INGSA, in New Zealand, because they know the statistical concepts but also understand the kinds of questions that policymakers have.

Showing people that even data scientists themselves were disagreeing over definitions has actually made the guide more engaging for some people. I couldn’t tell you how long we spent agonizing over the difference between an algorithm and a model. If the experts can’t agree on the precise academic definition, let’s give the public a definition that shows them what matters in terms of how we then make decisions based on the outputs of the model or algorithm.

Emily Jesper-Mir and Tracey Brown OBE of Sense About Science with Elsevier Foundation Director Ylann Schemm at Evidence Week in the UK Parliament.

With that in mind, the guide being about conversations rather than pure statistics, how have you struck a balance between providing a sufficient level of detail without hindering potential for discussion?

TB: The project originated in years of questions we’ve been brokering between the public, politicians and journalists, on the one hand, and the research community on the other. That’s why we understood that what was needed was not a guide to statistics but a guide to conversations about the implications of statistics.

We began work by using the questions from projects we’ve worked on such as helping parents navigate the outcomes of pediatric surgery and creating user-led presentations of environmental monitoring. We then user-tested our first attempts to break the conversation about quality in data science into three recognizable questions.

In the end it comes down to what people actually need to know in order to challenge decision-makers about quality and reliability in their use of data. These are: Where does it come from? What is being assumed? Can it bear the weight being put on it?

ER: Yes, you don’t need to be an expert to start that conversation. I could ask a medical researcher how many patients were in her trial testing a new cancer treatment without knowledge of oncology. There are some fairly complex concepts that the guide touches on within those three fundamental questions, such as the trade-off between sensitivity and selectivity, data dredging or overfitting, but readers don’t need to become an expert to bring these up in a conversation or to know that they are important.

We were careful, wherever these more difficult concepts were described, to use really relatable examples to anchor these in terms of their real-world consequences to motivate readers to grasp their importance. For example, the fact that overfitting may be partly to blame for the energy accident at the Fukushima nuclear plant, which you know well. Or how important prediction intervals are when interpreting political polls.

I’ve seen the finished guide. You’ve really managed to create something unique – information collated in a way that is otherwise not available to people. Who are you hoping this guide will reach?

TB: What we are aiming for is that everyone who might want to reach for this guide can find it.Our starting point is to take it back out through the networks that were involved in creating it. That’s why we are launching it first at the 2019 INGSA conference in Dublin on 27 June, and with all the data science networks and public groups we involved in preparing it. It’s also why we are encouraging Elsevier and every organization involved in creating data tools to help us launch it in many different parts of the world. We are happy for people to take the content and reshape it for themselves. The key question is that they do something – take responsibility for equipping our societies to ask these crucial quality questions.

An infographic from the guide. You can <a target="_blank" href="">download the full guide here</a>.


Written by

Anders Karlsson, PhD

Written by

Anders Karlsson, PhD

Dr. Anders Karlsson is Vice President of Global Strategic Networks at Elsevier. With Tokyo as base, he covers the Asia Pacific Region. He has a background in science diplomacy, having headed the Embassy of Sweden Office of Science and Innovation in Tokyo for five years, as well as in academia, having served as Professor of Quantum Photonics at the Royal Institute of Technology – KTH in Sweden for 10 years. He has a PhD in Electrical Engineering, and a MSc in Engineering Physics, also from KTH. At Elsevier, he has been engaged in a variety of projects, such as Elsevier’s reports on Sustainability Science,  Disaster Science and Artificial Intelligence . He is a member of Elsevier’s sustainability board as well as the Vice-Chair of the Japan branch of the International Association of Science, Technology and Medical Publishers – STM.


comments powered by Disqus