Back in November, I co-hosted an unusual workshop at the International Network of Government Science Advisors (INGSA) meeting in Tokyo. Sense About Science – an independent charity Elsevier partners with to promote awareness of research quality and peer review among the public, journalists and policymakers – wanted to test a framework for a public guide to data science.
With daily headlines touting the promise of big data and AI in predicting everything from the path of the next epidemic to end-of-season league positions of football teams to who is most likely to commit a crime, Sense about Science was asking what people needed to know in order to evaluate these claims.
The organization’s mission is to promote and defend the public interest in sound science and evidence. For Elsevier, as one of the largest providers of curated scientific content, data and information analytics, the project seemed a natural fit.
The project stemmed from a roundtable at the Broad Institute in Cambridge, Massachusetts. At Sense about Science’s request, Elsevier colleagues there hosted a small meeting with thinkers and statisticians from leading US institutions about the kinds of insights the public needed on data science. INGSA in Tokyo provided an opportunity to try them out.
I got the point. My own background is in engineering physics. When working at the Swedish Embassy in Tokyo as Science Counsellor, I was involved in our embassy’s reporting on the March 2011 Triple Disaster and notably the Fukushima Nuclear Accident. Official information was sparse and at times not well communicated – for example, radiation levels were given without any baseline on whether they were actually dangerous. That overall experience was an eye-opener as to how a guide to data science could support a better dialogue between decision makers, researchers and the public.
It was a huge task – to create a simple opening into an emerging science that could be relevant for people reading claims about Facebook data in the news as much as for government procurement officials buying in data services for health. Our Tokyo workshop, chaired by Prof. Yuko Harayama, proved that this was an important responsibility and that the challenge is global. It also showed that steering people through questions of quality and reliability in data science analysis was a daunting journey. But, eight months later, they are ready to publish.
When I met you, Tracey, you were frustrated about the lack of resources available to help people get involved in the conversations happening around data science.
TB: I was. I was particularly frustrated about discussions in Europe, where leading figures in policy and science were speaking at conference after conference despairing about the lack of respect for evidence and the rise of a ‘post-truth’ public. At the same time, they were announcing millions of euros on 'big data strategies' and hadn’t given thought or energy to asking whether people are equipped to ask the fundamental questions about quality. Here was an opportunity to equip citizens to be critically engaged in the evidence and decisions shaping their lives, and the people who could be leading that were too preoccupied to see it.
You were already taking matters into your own hands when we met in Tokyo. Tell me about what has happened since then. Errin, you’ve had the task of marshalling data scientists to the cause?
ER: We have had some excellent input from researchers in different fields of data science. It was a bit of a battle for simplicity. And often a battle between different experts about precise definitions or the need to add lots of detail to simple statements. This rather missed the point.
How did you overcome that?
TB: In our discussions with data scientists, we challenged them to cut to the heart of an issue and drop as much technical language as possible. This was helped by getting across the idea to data scientists that we weren’t trying to write a data science textbook but to equip people with the language and basic concepts that would empower them to start a conversation about what weight different data analyses could bear.
ER: It has been hugely helpful for us to discuss it with people like Dr. Brad Fenwick (Senior VP for Global Strategic Alliances at Elsevier), Dr. Michiel Kolman (Senior VP for Global Academic Relations at Elsevier) and some of the team who work with Prof. Sir Peter Gluckman, chair of INGSA, in New Zealand, because they know the statistical concepts but also understand the kinds of questions that policymakers have.
Showing people that even data scientists themselves were disagreeing over definitions has actually made the guide more engaging for some people. I couldn’t tell you how long we spent agonizing over the difference between an algorithm and a model. If the experts can’t agree on the precise academic definition, let’s give the public a definition that shows them what matters in terms of how we then make decisions based on the outputs of the model or algorithm.
With that in mind, the guide being about conversations rather than pure statistics, how have you struck a balance between providing a sufficient level of detail without hindering potential for discussion?
TB: The project originated in years of questions we’ve been brokering between the public, politicians and journalists, on the one hand, and the research community on the other. That’s why we understood that what was needed was not a guide to statistics but a guide to conversations about the implications of statistics.
We began work by using the questions from projects we’ve worked on such as helping parents navigate the outcomes of pediatric surgery and creating user-led presentations of environmental monitoring. We then user-tested our first attempts to break the conversation about quality in data science into three recognizable questions.
In the end it comes down to what people actually need to know in order to challenge decision-makers about quality and reliability in their use of data. These are: Where does it come from? What is being assumed? Can it bear the weight being put on it?
ER: Yes, you don’t need to be an expert to start that conversation. I could ask a medical researcher how many patients were in her trial testing a new cancer treatment without knowledge of oncology. There are some fairly complex concepts that the guide touches on within those three fundamental questions, such as the trade-off between sensitivity and selectivity, data dredging or overfitting, but readers don’t need to become an expert to bring these up in a conversation or to know that they are important.
We were careful, wherever these more difficult concepts were described, to use really relatable examples to anchor these in terms of their real-world consequences to motivate readers to grasp their importance. For example, the fact that overfitting may be partly to blame for the energy accident at the Fukushima nuclear plant, which you know well. Or how important prediction intervals are when interpreting political polls.
I’ve seen the finished guide. You’ve really managed to create something unique – information collated in a way that is otherwise not available to people. Who are you hoping this guide will reach?
TB: What we are aiming for is that everyone who might want to reach for this guide can find it.Our starting point is to take it back out through the networks that were involved in creating it. That’s why we are launching it first at the 2019 INGSA conference in Dublin on 27 June, and with all the data science networks and public groups we involved in preparing it. It’s also why we are encouraging Elsevier and every organization involved in creating data tools to help us launch it in many different parts of the world. We are happy for people to take the content and reshape it for themselves. The key question is that they do something – take responsibility for equipping our societies to ask these crucial quality questions.