As a university’s chief research officer, Dr. Brad Fenwick faced a monumental challenge: tasked with coordinating and directing the allocation of funds to support research, he quickly realized that something crucial was missing.
“My frustration at every step was in not having good data on which to make big resource allocation decisions,” he recalled. “Should we build and equip a $30 million dollar building hoping that it would produce something – with no real strong evidence that it would? As an alternative, we could hire prized faculty and strengthen our retention efforts – or buy unique equipment. All these things showed promise, but we still didn’t know which, or which combination, of these would actually be best related to increasing reseach productivity and success.
“I saw time and time again that decisions were made with essentially no data – mostly on political pressure, on gut instinct,” said Dr. Fenwick, now Senior VP of Global Strategic Alliances at Elsevier.
His foray into the “dark side” led him to a realization: “Maybe economics could help higher education leaders make wiser decisions with limited resources.”
Dr. Fenwick’s realization was the impetus behind events he co-organized at the 2018 AAAS Annual Meeting. An interdisciplinary panel and workshop brought together eminent scientists and economists to talk about challenges to scientific productivity and effectiveness – what causes them and how to address them. The events were filled to capacity, and conversations continued long after they concluded. The event was co-organized by Drs. Marie Thursby, Jerry Thursby and Karim Lakhani in cooperation with AAAS, with support from the Alfred P. Sloan Foundation.
In a panel on the Optimization of Scientific Performance Across Disciplines, professors from five US universities talked about how science is conducted in their fields. They delved into topics such as how labs are organized, how research is structured and managed, how productivity is measured, how they collaborate and share information, how funding and other resources are allocated across the sciences, and how knowledge is shared and commercialized.
“We are here to talk about optimization of scientific effectiveness. What does that mean?” asked moderator Dr. Marie Thursby, Regents’ Professor Emeritus at the Georgia Institute of Technology, who has published extensively on the economics of innovation.
“It’s a very important question,” she said. “You hear a lot about it because of low funding these days; people complain about it all the time. But we have a hypothesis that the real problem with science – or the worst problem with science – is not the level of funding but the way funding is allocated.”
Economists can provide insights on allocation, she added, but to do so, It’s important to have data on how science is produced.
Data was a common thread throughout the day, and for most, funding was closely tied to access to data and the ability to make use of it.
Big telescopes mean big science – but is that better?
Dr. Amy Barger, Vilas Distinguished Achievement Professor of Astronomy at the University of Wisconsin-Madison, pointed out that the drive for cost efficiency – and the greater buying power of larger universities – is changing the way astronomy is practiced.
As an observational astronomer, she spends much of her time collecting data on large telescopes around the world.
“We have a slightly different model than a lot of other scientific disciplines in that we don’t require our own personalized fancy labs when we arrive at the university,” she explained. “But we have to have our universities invest in large telescope facilities over the long term, so it’s a lot of expense, both in the initial construction of the telescopes and then in the operating costs.”
That reality has led to the construction of dedicated survey telescopes, which is changing the way data is collected and used in research. Instead of individual researchers, or small groups of researchers, coming up with their own projects, collecting their own data, analyzing it and writing a paper, the trend is shifting to a “big science approach.”
Dr. Barger mentioned the large synoptic survey telescope being built in Chile. “It will survey the sky every three nights, so the amount of data coming out of that system is tremendous,” she said. “And it’s a huge challenge just figuring out how you are going to deal with that kind of data flow.
“But also it changes the research that’s being done because now you’re starting from a data set that already exists; you’re not designing your data set and coming up with (your own) problems. So I think the tendency then is to start having large collaborations, which (is) really changing the character of astronomy.”
It can lead to collaborations that are “forced” and less than friendly. But beyond that, it can take away from the more individual approach to science.
“To me there is a real loss in not having the students be there from the beginning, where they are completely in charge of the project and getting their individual data,” Dr. Barger said. “I feel like large collaborations work very differently – you start to become just part of the wheel, and you’re no longer in charge of the whole thing.”
Meanwhile, another trend is widening the gap between top-tier universities and those with less generous endowments. “Astronomy has moved from national facilities that everyone has access to, to much more elite universities buying into the latest, greatest telescopes – and other places being left behind,” Dr. Barger said.
“Science is becoming a question of the haves vs. the have nots”
That concern was shared by fellow panelist Dr. Gregory Petsko, the Arthur J. Mahon Professor of Neurology and Neuroscience at Weill Cornell Medical College.
“Science is, in this respect, resembling American society in that it’s becoming a question of the haves versus the have nots,” he said.
As an example, he referred to the US National Institutes of Health (NIH). “The NIH budget doubled over a period of less than 10 years around the turn of this century, and the striking consequence of that was that very few new investigators were added to the pool as a result of that increase,” he said. “So what happened to all that money? Well, most of it went to big labs that got bigger — and this is not an optimal way of doing science.
“You don’t want science to resemble those hippie maps of the United States from the 60s that had the East Coast and Berkeley and nothing in between – and yet we’re approaching that at an alarming rate."
One of the consequences of this has been the de-emphasis on basic research funding, he said. "It’s important to remember that most great science wells up from the bottom, and that top-down projects organized by groups of individuals or by government agencies for particular purposes only work well when they’re engineering projects.
“The small labs, where a lot of the great basic research gets done, are having increasing trouble finding the money they need (because) a lot of the funding has been diverted to big labs and big projects. I don’t think that’s healthy. … And I think something needs to be done about it.”
Are we giving money to the right people?
How science is funded was also an issue for Dr. Eva Guinan, a Professor at Harvard Medical School and Director of Translational Research, Radiation Oncology, at Dana-Farber Cancer Institute. We give grants to people who have absolutely no training in how to manage money or an organized enterprise,” she said. “And I can’t really think of a place that you would do that otherwise.”
We give large amounts of money to scientists who have no training in how to manage personnel, how to select personnel, how to manage to a budget, how to think about system constraints, how to do anything else managerial. And then we’re surprised when things don’t move forward with any sort of efficiency and when we can’t actually compare one output to another. So I think that there’s a, a failure of infrastructure, if you will, in the broadest possible construction of infrastructure within the laboratories. It contributes to global inefficiencies and then those inefficiencies get read as failures of the (specific experimental) system and of the science, where that may not necessarily be true.
She said she has observed inefficiencies in her work with multidisciplinary and translational research. “I’ve had the good fortune … of taking things that we developed in our lab, or in our lab with collaborators, all the way through from the bench, or the first observation, through the FDA, to drug approvals doing so-called translational science. And in the context of doing that work, I became really very interested in – and irritated by – the inefficiencies and randomness of that transition to the clinic and the inwardness of the process which seemed really quite unnecessary.”
As a result, Dr. Guinan started collaborating with colleagues at the Harvard Business School, including Dr. Karim Lakhani, who presented in the workshop that followed. Together, they’ve been exploring whether there are insights or practices they could derive from economics and other social science disciplines that could improve processes in the medical field.
The panel, drew about 100 people, including academic researchers and economists.
“Then a miracle occurs”
In a subsequent workshop organized by Dr. Fenwick, economists and scientists from academia tackled the topic of “Researching Scientific Production: Where are we? Where do we go from here?” They discussed the challenges facing scientific productivity and effectiveness and how the system could be fixed.
With wit and ingenuity, Dr. Jason Owen-Smith, Professor of Sociology at the University of Michigan, focused on how data can be useful, specifically the data repository of the Institute for Research on Innovation & Science (IRIS), of which he is Executive Director. With IRIS, member universities send record-level administrative data, which IRIS links to public data. The goal of IRIS is to enhance our ability to understand the value of research to the public. As a shared resource, IRIS can also lower the cost of getting access to data, cleaning it up and making it available.
Dr. Owen-Smith pointed out that in 2015 – the year IRIS was formed – research universities invested about $214 in academic research for every person in the United States, 55 percent of which came from federal funding. “We make those investments as a society to improve the state of human knowledge … with the promise that somehow this will improve the quality of life for our nation and the world,” he said. “And we do a fairly bad job of explaining how that happens. A lot of our explanations come down to: ‘Fund the super-conducting super collider, and you’ll get the next World Wide Web. It’ll be great.”
He then showed a New Yorker cartoon of two scientists plotting out an experiment at a blackboard. Scrawled in the middle, flanked by detailed equations, were the words “Then a miracle occurs.” The caption: “I think you should be more explicit here in step 2.” As Dr. Owen-Smith explained:
There are inputs and there are outputs, and in the middle, a “miracle” occurs. Something needs to be done with step 2. So part of what we’ve been trying to do is to think about ways to systematically fill in step 2 by collecting and creating a data set that can be responsibly shared as broadly as we can with a research community, like say this one, in order to support good research and reporting that can be cumulative and speak to each other, can be replicable, and can, for lack of a better term, help us understand, explain and eventually improve the public’s view of the value of the research that’s being done.
“Science investments flow into the university from a variety of sources,” he continued. “We tend to get stuck talking about expenditures. So what IRIS tries to do is to look at how investments and spending enable work. They enable people, investigators like us, to hire the people and buy stuff necessary to get the productive work of research done. Hiring people and buying stuff has near-term stimulus effects that we can measure with some of these data – but that’s not the purpose of science, certainly.”
Of course, the fact that it’s not the ultimate purpose of science is not lost on the public and policymakers.
“The predominant image is that grants flow to someone like me, and I use them to put on a white lab coat, drink good lattes, and take the summer off,” Dr. Owen-Smith said. “That is obviously not the case, but because we haven’t been able to specify the human capital and input sides of the production function as much as we might like, we have trouble saying that. The work produces a bunch of good things – discoveries, patents, publications, tangible and intangible property. Critically, it also produces trained, skilled people who leave the university and take their skills elsewhere. ... And we think that the movement of people and skills out into the larger economy in society is what drives public value in a primary way. And we’ve not been able to see that as clearly as we’d like.”
Using data to show the real value of science was the main impetus behind IRIS.
Shaking up the system with data and economics
In his presentation on The Future of Academic Research, Dr. Brad Fenwick showed a slide with Warren Buffett’s “three I’s” – the “natural progression” of how good ideas go bad. Paraphrasing the billionannaire, Dr. Fenwick said:
The innovators see opportunities that others miss and champion new ideas that create value. The imitators copy what the innovators have done; sometimes there are marginal improvements, but mostly the unique benefit is lost. And then we have the “idiots” out there that undermine the innovation that’s going on.
All to frequently, he said, promises and policies are not based on solid scientific evidence.
He went on to talk about Elsevier can help make the scientific process more efficient through its tools and services as an information analytics company. He pointed out that Elsevier’s extensive scientific research and health data can be especially helpful when combined with legal and risk data from LexisNexis and public data sources like patent records.
“While many people know Elsevier as a large publisher, we’re a big data information company,” he said. We help research institutions and research professionals produce and use data to make better decisions so they’re more productive.”
He pointed out that research and development is becoming an increasing complex business, prompting players to become “hyper-competitive,” often to their own detriment. “Increasingly, interdisciplinary research is going global, and people don’t want to accept that,” he said. “There are increased institutional expectations – always do more, always do better – and there’s lots of pressure with the potential for multiple points of failure. And we don’t know why things fail; I would have programs that failed and not be able to unpack it or understand why, so we couldn’t correct it going forward.”
The pressure is compounded by regulations and scrutiny, an increase in reporting requirements, and management and leadership challenges that are often underappreciated, he said.
Meanwhile, in academia, budget constraits have become a reality, and academic executives have accepted that funding is unlikely to increase. Ultimately, he said, all this has led to “growing levels of frustration with no easy solutions.”
For Dr. Fenwick, this dilemma was at the heart of the events that day and his call to have an ongoing collaboration among scientists and economists. For academic research to continue to progress, the system must work a bit more efficiently – and that’s certainly a possibility.
“My sense of your group is tremendous in terms of helping us get to these levels,” he said. “Let’s address the frustrations and try to come up with some solutions through data.”
When questions outnumber answers
Dr. Karim Lakhani, Charles E. Wilson Professor of Business Administration at Harvard Business School, said there were still many questions to be answered in the realm of scientific productivity. To investigate this topic, he approached Dr. Guinan at the Harvard Medical School and Dana-Farber Cancer Institute.
“It occurred to us that the Medical School is actually a great laboratory for us to study innovation,” he said. “We’re blessed with an awesome hospital, lots of scientists, lots of great data.”
They Laboratory for Innovation Science at Harvard (LISH), which he now co-directs with Dr. Guinan, is funded in part by the Sloan Foundation to do deep dives into questions of scientific production.
“Our motto has been to help solve innovation problems,” Dr. Lakhani said.
He said they delve into fundamental questions like:
How do scientists choose topics to pursue? What constitutes a portfolio? …How are labs organized and managed? What are the management practices to running a lab? Are they effective? How do we think about effectiveness?
A key question has to do with how to identify salient outcomes: “I don’t think we have agreement on salient outcomes. …You might measure publication, you might measure patents, you might measure citations. … But what are the relevant outcomes? And how do we actually think about outcomes across the production process?
“Part of the challenge we’ve had in running experiments with the medical school has been, well, I make an intervention, and if my outcome variable is a publication, well that’s going to be months or years away. Nobody’s going to be able to hang out and wait and hang around that long for results. So how do we think about additional sort of surrogate markers in the scientific production process that helps us think about the final outcomes? And what are the relevant outcomes that we care about?”
He also raised the issue of competition vs. collaboration, mentioning a colleague in attendance: Dr. Paula Stephan, Professor of Economics at Georgia State University and author of the book How Economics Shapes Science (Harvard University Press, 2012)
There’s a Mertonian view of how amazing science is – that it’s a Kumbaya, let’s hug together community, it’s awesome, we share knowledge together. But as Paula has shown, as lots of people have observed, this is a brutal winner-take-all competition as well. A grant call is a competition. There are winners and losers across the board. So how does competition shape the production function? How is this competition defined? …
I don’t think we have the basic facts around these things. And also again, there’s this question around convergence and diversity. If we move to a collaboration model, then will there be any incentives to perform higher? Because one of the things of economics is that competition drives rivalry or drives performance. At the same time, it could be harmful. Think about the race to discover the virus for AIDS with two groups working independently and it took them a while to get there. What if they shared their findings? So basic questions we that don’t have answers for. Certainly this morning we talked a lot about shared resources, and if in fact lots of science is becoming capital-intensive, then who gets access? Who invests in these tools, and how do they get used?
Ultimately, he said, the question comes down to how competition in the face of limited resources shapes research-driven knowledge production and sharing. What role will a focus on enhanced efficiency rather than greater funding have on research productivity?
After his presentation, Dr. Marie Thursby, who moderated the session, gave a technical explanation of her view of the scientific production function.
Continuing the conversation
After the presentations and Q&As wrapped up, few people left the room. Most lingered to talk to each other about their work and challenges in the context of the workshop. At one table, an economist “interviewed” a research scientist at length to find out how the process worked at her university, from funding to equipment and staffing.
Dr. Stephan, who has written extensively about the economics of science, said she was encouraged to see so much interest in this topic. “I’m really fascinated with the idea of a scientific production function, I always have been, so it’s very encouraging to me to see that other people have become interested in it,” she said. “Because I’ve been fairly convinced that we’re not using resources as efficiently as we could.”
She said she was particularly interested in the role of laboratory equipment.
“In my book,” she said, “I think of universities as upscale shopping malls that are in the business of building research space and then recruiting scientists to fill it. And scientists pay to be there through salary buy-off and indirect costs, and universities are trying to maximize reputation and money by doing that.”
So is there a better solution? “Who knows? I think that’s a big issue here,” she said. “In the US at least, we just have one model of doing it. I don’t think we’ve thought about a better way. I think it’s very exciting to talk with other people and think about some of these issues.”
Dr. Maryann Feldman,the Heninger Distinguished Professor in the Department of Public Policy at the University of North Carolina, said she saw “great potential” in the ideas shared that day. In her research, she has focused on technology-based economic development and has written extensively on the process of commercializing academic research. She said her research team also found the need to examine earlier stages of the scientific process.
“We started out looking at technology transfer and the university licenses,” she said, “and then we realized that we needed to go further upstream to look at the funding of research and the organization of … production units or research groups.”
Dr. Guinan said the workshop gave her a broader appreciation of how different scientific disciplines work under divergent constraints as well as incentives. “We need to really think about whether we can generalize observations across them, and how we do that,” she said. “It both confirmed but really enlarged my understanding of that issue.”
Dr. Yasin Ozcan, a Postdoctoral Fellow at the Sloan School of Management at MIT focusing on technology management in health policy and innovation policy, said it was interesting to learn the particulars of the initiatives the speakers discussed. “Seeing all of this basically makes me think about alternative things I can work on down the road,” he said. “So as a junior member of the field, it was quite useful to me.”
“I think it’s an exciting time to be working on science policy,” said Dr. Bhaven Sampat, Associate Professor in the Department of Health Policy and Management at Columbia University and a Research Associate at the National Bureau of Economic Research. “We’re moving from sort of faith-based science policy, where the only thing anybody said was ‘We need more money’ and `Give scientists freedom,’ towards more appreciation of the need for systematic evidence. We’re able to test our long-standing assumptions about how science works and what types of policies and institutional structures work better at producing the outcomes society wants.
He said he found this movement promising for two reasons:
One is that I think both economists and scientists seem open to examining these questions rather than just assuming that the only thing science needs is more money. Second, there has been a data revolution, so there’s new data that Elsevier and others have that allow us to examine the scientific production function at a micro-level of detail that just wasn’t possible before. It’s really nice that there appears to a willingness at Elsevier to work with academics to make these data available, so it’s an exciting time to be in the field.