M T Fahey

A Post about Open Science and Incentives

In General, Science on May 25, 2011 at 9:04 pm

Slate Magazine recently published an excerpt from Tim Harford’s new book, Adapt: Why Success Always Starts with Failure. The selection recounts the story of researcher Mario Capecchi’s work on gene targeting, funded by the NIH despite strong recommendations from grant reviewers that he abandon the project in favor of less speculative work. His risk-taking was rewarded by a Nobel Prize, and his reviewers later wrote to him in apology: “We are glad you didn’t follow our advice.”

In defense of his book’s title thesis, Harford invokes an economics study on the effects of financial incentives on creative scientific achievement. The paper tests the theoretical framework suggested by Gustavo Manso (2009), in which particular incentive schemes increase the production of innovative ideas (in Harford’s words: “insanely great ideas, not pretty good ones”). According to the researchers:

“The challenge is to find a setting in which (1) radical innovation is a key concern; (2) agents are at risk of receiving different incentive schemes; and (3) it is possible to measure innovative output and to distinguish between incremental and radical ideas. We argue that the academic life sciences in the United States provides a near-ideal testing ground. ”

It was an ideal testing ground. The Howard Hughes Medical Institute (HHMI) investigator program, one source of funding, has long renewal cycles (5 years), a robust and detailed review process, and the stated goals to fund “people not projects” and to “push the boundaries of knowledge.” This leads to a system which gives investigators flexibility in the projects they pursue and time to invest in exploratory efforts. In contrast, the National Institute of Health (NIH) R01 grant program has a shorter funding period (3 years), a stricter review process with less feedback, and an emphasis on funding specific projects. By measuring the number of publications from each investigator that fell into the top citation percentiles, the novelty of the attached keywords (in relation to the entire body of literature and the researcher’s past work), and the range of journals that cited the work, the researchers found evidence that HHMI investigators produced more novel research and more high-profile papers (while also producing a greater number of flops) than NIH controls.

The researchers are careful to note that their findings “should not be interpreted as a critique of NIH and its funding policies.” After all, researchers that are awarded HHMI grants (and the NIH MERIT controls) have been judged to exhibit extremely high potential. The investment in time (detailed project reviews) and risk (a high degree of freedom and low expectations for immediate results) for the program suggests that it may not be beneficial to replace the NIH incentive structure with HHMI-like criteria. The ideal situation may be closer to the one currently in place — a combination of the two incentive systems. A more project-based system may be better for younger, less experienced scientists, while the HHMI system would be best reserved for the select group who would flourish with less constraints.

How else might small differences in incentive structure determine the quality of scientific output? What implications does the confirmation of the Manso model have for the future direction of scientific investigation?

The incentive structure in scientific research is likely to change drastically in the next decade. Like the field of journalism (another information-gathering and sharing enterprise), the practice of science is poised at the edge of an internet-driven revolution in connectivity. Driving this change are journals like the Public Library of Science (PloS), which makes scientific papers available to the public via the internet using an author-pays model, and groups like openscience.org, which encourages the free sharing of methodologies and datasets with the public. Some major academic publishers are also moving to a hybrid system in which authors can pay a fee to have their paper publicly available, and groups like HHMI have agreed to cover the extra charge for their grantees.

The poster child of the open science movement is the Human Genome Project, which sequenced and published the entire human genome in 2003, in the process narrowly routing a proprietary attempt to do the same. Following in the footsteps of that success, all DNA sequences generated by recipients of NIH grants are required to be entered into the GenBank database, creating a massive searchable repository of our genetic knowledge. Open science has the potential to help scientists wade through massive amounts of data, to find patterns in extremely complex systems, and to substantially increase the speed of scientific progress.

Despite the success of these open science experiments, scientists are understandably slow to switch over. Dan Gezelter at openscience.org writes:

“Right now, the incentive network that scientists work under seems to favor ‘closed’ science. Scientific productivity is measured by the number of papers in traditional journals with high impact factors, and the importance of a scientists work is measured by citation count. Both of these measures help determine funding and promotions at most institutions, and doing open science is either neutral or damaging by these measures. Time spent cleaning up code for release, or setting up a microscopy image database, or writing a blog is time spent away from writing a proposal or paper. The ‘open’ parts of doing science just aren’t part of the incentive structure.”

Time investment is not the only reason the scientific community is resistant to moving to the open system. Opening up your data to analysis by other scientists is necessarily risky, and not something scientists are likely to do en masse until it is fully expected by the community. It involves massive reform of the scholarly publication system, as well as the reputation bookkeeping done by that system. It requires researchers be comfortable with managing their online identity and with using new software tools. It will involve sweeping changes in the way science is reported.

An online publishing system, especially one that encourages self-publishing, would allow scientists to publish smaller chunks of work at a time. Results from individual experiments could be made available as soon as data is gathered. Data analysis and interpretation could become a much more open process. The smallest publishable unit would shrink, allowing researchers across the world to build on each other’s findings in a fraction of the time it currently takes. Many of the publishing biases that plague the current system (eg unwillingness to publish replication of experiments or negative results) could be eliminated.

It is possible that the ability to publish in this manner, with atomized units (maybe single experiments), would lead to the expectation that scientists publish much more frequently. Having shorter periods of expected achievement would mimic the incentive structure of the NIH rather than the HHMI; in the same way that shorter review periods prevent exploration, expected rapid publication of data on the internet could incentivize less risky experimentation.  In this case, perhaps an increase in HHMI-like funding incentives would be appropriate.

There is a sense in which faster publication in smaller units could cheapen the value of a publication. There would be less effort, less time, and less reputation at stake for each piece released. The current system, with the huge time investment necessary to get a paper published in a traditional journal, encourages lumping of many experiments and forces researchers to piece all of their work together into a convincing narrative.  A scientific paper today may span many years of work. It puts all of a researcher’s eggs into one basket.

Rapid publication of open data prevents researchers from shuffling their experiments to tell a better story. It makes it difficult to change how data is perceived by changing the way it is presented. Peer review could be done online, transparently, by a larger group of people. The shorter time between finishing and experiment and publication would automatically help put experiment in context with other work being done. Review and meta-analysis would be more important than ever to retain the continuity and coherence of the scientific narrative being created (this could be a new role for journals as they lose their base of readers and contributors).

The face of science could be changed for the better, medical advances could be faster, and public understanding of the sciences could be improved, and it could all happen very soon if institutions with pull in the scientific community (the NIH, HHMI, journals, and universities) created the right incentives.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: