The Canadian Institutes of Health Research and its partners have so far handed out more than 550 grants worth over $250 million for research related to COVID-19. Some will undoubtedly lead to significant breakthroughs that will help us deal with this pandemic and those that follow in future. Others, inevitably, will fail to live up to their potential and lead nowhere. So Canadian researchers are beginning to ask how those hundreds of grants will be evaluated to ensure that the money was well-spent.
“Is there going to be any follow-up?” asked Paola Marignani, a molecular biologist at Dalhousie University. The funding councils examine a researcher’s progress and productivity when they apply for grants to be renewed, said Dr. Marignani. But most of the COVID-19 grants are likely to be one-offs, with the researchers going back to their previous projects once the pandemic funding ends. “We know the work is being done now, but will we ever know if the widget was produced?” she asked.
Abraham Fuks, an immunologist at McGill University, said it is important and necessary to evaluate how the money was spent in the spirit of appreciative enquiry – looking both for what went right, and what could be done better. “We all acknowledge that the people who made the decisions were highly skilled, and did so in good faith under crisis conditions,” he said.
Dr. Fuks said two different kinds of reviews are needed. First, a granular evaluation of the outcomes of the projects that were funded, similar to what is done for project renewals, as a kind of retrospective peer review. “I’m concerned that some of the peer review done last year in haste was not as useful as it could have been,” he said.
Second, a higher-order evaluation should be done of how the emergency grant programs were rolled out to look at whether there was sufficient coherence in their goals and governance, and coordination between the myriad funding agencies and partners. “Were there ways to simplify the process?” asked Dr. Fuks. This exercise will help ensure that the response to the next pandemic can be rolled out more smoothly, he said.
Read also: Was COVID-19 research funding well spent?
Adrian Mota, the CIHR’s associate vice-president for research programs, said that all recipients of the emergency COVID-19 funding will have to submit an evaluation of their projects when they finish, and the evaluations will be more detailed than for a standard project renewal. “It’s important to understand what we got out of these investments, what did they yield and who was involved,” said Mr. Mota. “The real-world impact of research can sometimes be pretty tenuous, but we might be able to see it a bit more clearly this time because of the shorter timeframes.”
The CIHR has already started evaluating some aspects of the rollout of the extra funding, said Mr. Mota, and made changes to improve things as it went on. After the first round of grants, in February 2020, the agency saw that there were fewer applications from women than expected. Concerned that the extremely short deadlines for that competition may have made it more difficult for women who were dealing with the lion’s share of the pandemic’s disruptions to their home lives, the CIHR extended the deadlines for future rounds and saw a better proportion in the next competition.
One thing that the agency will be examining closely is how to build on the research and networks that were created to deal with this pandemic, in order to better prepare for future ones. “The investments in SARS and MERS kind of died out,” said Mr. Mota. “If the work had been sustained, we might have been better prepared this time.” The CIHR’s new centre for research on pandemic preparedness and health emergencies is one means of ensuring that doesn’t happen again, he said.
Nathalie Grandvaux, a virologist at Université de Montréal, said that although most of the COVID-19 research grants were intended to be for short-term, one-off projects with a relatively quick payoff, a bigger sign of success will be whether any of them develop into continuing research programs. “If this investment doesn’t help develop long-term projects, then it is kind of lost,” she said. “If there is nothing long-term, the money would have been better spent on basic research.”