For a researcher, the publication of an article can look like the end point in a long process of hard work. But what happens afterwards – presentations to peers, media coverage, citations – is crucial to determining the significance of a study’s findings.
Qualitative and quantitative metrics have brought us a long way in terms of evaluating research output. But how can we measure the impact research has on society?
To help us answer that question, the startup Newsflo joined Elsevier in 2015, expanding our measurement capabilities with a tool that monitors researchers’ impact across a broad range of media sources. Today, a small team of two software engineers and a product manager is working with our technology teams to optimize the infrastructure and integrate Newsflo into some of Elsevier’s flagship products.
“So much has happened in the past year,” said Product Manager Dr. Elaine van Ommen Kloeke, who is based in Amsterdam. “Before, I worked as a publisher, so the engineers were a bit suspicious when I joined. But I just threw myself in: I took trainings and spent as much time as possible with the engineers in London. We still call every day.”
It paid off. The team worked on making the infrastructure stable and integrated the service into several other Elsevier solutions.
Software engineer Stuart White, who works in Elsevier’s London Wall office, faced his own challenges when he joined last year. “I was new to big data and machine learning, and that was one of the main attractions for me in this job,” he said. “Technology and tools are changing every year, so there’s always a steep learning curve when you join a new project.”
Together with his teammate Kyriakos Vallianatos, Stuart is in charge of making sure Newsflo is up and running and looking for ways to add new features more easily. They are also expanding the utility of Newsflo by integrating it into different platforms. So far, Newsflo’s media mentions can be found on Elsevier’s SciVal, Mendeley and Pure.
How Newsflo works
Newsflo aggregates content from more than 45,000 English-language news outlets in over 20 countries, in collaboration with Elsevier's sister company LexisNexis. It then uses data from Scopus, the world’s largest abstract and citation database of peer-reviewed literature, to filter out those articles about research. This filtering is done using a text-tagging technique that identifies individual researchers through their affiliation.
Researchers can check media mentions of their work on their personal profile on Mendeley. Newsflo can also present media mentions at a university or funding body level.
Collaboration is key
Being part of a small team enables them to work in a highly focused manner. “We are always looking for pragmatic, quick wins,” Stuart said. “The size of our team allows us to be more agile and more experimental.”
Meeting with different colleagues is important, too, since services like Newsflo become a lot more useful when they are combined with other tools and data.
Most recently, a data pipeline was set up between Newsflo and another startup that recently joined Elsevier, Plum Analytics, which offers a similar service through a different methodology. Whereas Newsflo identifies mentions of research by looking for researchers, Plum searches for references of research articles through their DOI or URL. This can lead to different results, so sharing data with each other helps improve both platforms.
Last year, they teamed up with SciVal, a benchmarking tool that provides easy access to the research performance of 8,500 research institutions and 220 nations. As a result of the collaboration, SciVal now has a feature that can present media coverage of research at a university level, with additional metrics to identify mass media outlets.
Room for innovation
Working agile through continuous improvements helps Stuart, Kyriakos and Elaine make sure they deliver value to the users of Newsflo. Since there are small feedback loops, priorities can be shifted quickly when necessary. “I went from being a bit skeptical about Agile to a strong believer and advocate of this way of working,” Stuart said.
Innovation also happens at the London office’s monthly hack days, where tech staff are free to work on any kind of new idea or project. For the Newsflo team, it means they can experiment with new features that sometimes even make their way into the product. Last year, they laid the groundwork for a newsfeed streaming service that lets users see real-time media mentions of their research. More recently, Stuart has been looking into using machine learning and smart algorithms to identify different types of media mentions to make Newsflo’s results more accurate.
These kinds of experiments are what make the work so interesting for Stuart:
The landscape is changing around you rapidly. Through machine learning, technology can have a much bigger impact on people’s lives than I thought was possible before. It’s kind of a force multiplier rather than a linear improvement. Developers are much more empowered these days than they were when I started out, and you need to be a lot more knowledgeable about a lot wider areas.
The rise of machine learning is opening opportunities for the team to bring structure in the hundreds of thousands of media articles Newsflo receives on a daily basis. Stuart has been using the hack days to find ways to identify articles about science – and articles that should be excluded, like an article about a university football team or other unrelated news in which researchers are mentioned alongside their affiliation.
Elaine is already thinking about the next Elsevier platforms to integrate with: "It would be fantastic if we could expand to a platform like ScienceDirect, where we can really make a difference for a lot of people."
All in all, Elaine says she is happy in her new tech environment.
It is just so interesting and fun, and wonderfully geeky from time to time. I love that I am building something from the ground up, something that’s cool and new, that surprises people and makes them want to work with us.