In a perfect world, researchers develop flawless plans that sail through the ethics boards, attract committed and enthusiastic participants, the data fits neatly and logically together, and projects are created on time. And yet as most of us know, the term “perfect world” is a bit of an oxymoron.
So why don’t we talk more about potential failures in research?
A “grant from hell” inspired David Streiner and co-editor Souraya Sidani to call out to researchers around the world to share their experiences. The resulting essay collection When Research Goes Off the Rails: Why It Happens and What You Can Do About It was published in 2009 and covers stumbling blocks at every point, from ethics approval to data collection to collaboration. It not only shows how failure is part of research, but provides inspiration from researchers who overcame roadblocks to complete successful studies.
University Affairs: Why do you think there is hesitation to share research failure along with success? Before your book, where was this kind of information shared?
Dr. Streiner: If it came up at all, it was while you were having coffee with your supervisor, so I don’t know that anyone had ever written about it before. But everybody knew about it. Everyone who’s ever done research knows. You talk to your colleagues and supervisors and find out about it.
I had given a number of talks on the topic, but I was apprehensive talking to grad students. I thought I would scare them off research. But the universal response was that “we know this happens but nobody ever tells us, and we’re so happy that people are finally admitting what happens.”
University Affairs: Where do you think researchers should first learn about the realities of derailments? Would you like to see some of these potential derailments incorporated into research methodology classes?
Dr. Streiner: Absolutely. We aimed the book as supplemental reading in research methods classes. I don’t think the news should be given on day one, but it should be given before they get their degree. They shouldn’t be discouraged about it, because even with all the pitfalls, research gets done, and there are ways to overcome it and get around it. Isaac Asimov, the great science fiction writer, said, “The best words in research are not ‘eureka,’ but ‘that’s funny.’” And it’s things that turn out not quite as you expected that lead to the rarities of research.
University Affairs: Are there any areas where derailments can be more serious than others in terms of their impact on research?
Dr. Streiner: Probably the two most important areas are the design of the study and the source of the participants. This is a major problem in biomedical research, because drug trials are very well designed, but the first thing you read about in a drug trial is a long list of inclusion and exclusion criteria. It’s easy to understand from a researcher perspective why that is as it makes the results so much easier to interpret, but the patients in the study become less and less like the patients in real life. The clinician in his or her office doesn’t have the luxury of saying, “I’m sorry you have another disorder at the same time, don’t darken my doorstep again”, or “you only meet five of the six criteria, so I can’t treat you.” Reality is a lot messier than the study and that’s a major source of bias.
University Affairs: Gatekeepers, whether ethics boards or within the community to be researched, seemed to be a major roadblock in many essays—what role do these play in derailments?
Dr. Streiner: Talk to anyone who’s done research and the first thing they’ll complain about is the research ethics board, and I’m very sympathetic to that because I’ve had run-ins with REBs and also chaired the board for 15 years at McMaster and for ten years at Baycrest, so I know both sides of story. To an outsider, I know they can appear capricious. And some of them are; I’ve had my own fair share of problems. The problem is you don’t want to take away their autonomy. But ethics boards need to be aware that researchers are on a timeline: you get your grant and you have a year to carry out your research, so you can’t have it sit for six months.
University Affairs: What about other gatekeepers?
Dr. Streiner: Non-research organizations, school boards, Native groups, ethnic communities are where you get the biggest pushback because they have very little investment in research. Researchers don’t help when they don’t give back and make the community realize why the research is important, and how they can benefit. Giving back doesn’t have to be time consuming or expensive, it’s the token aspect of it, to show we’re grateful and we’re going to give something in return. Researchers have to realize that it takes a lot of groundwork and preparation. You can’t design a research project and then go to a community and say, “I want to study you people”— those people have to be involved at the very beginning.
University Affairs: How did people react after the collection came out?
Dr. Streiner: A lot of people have asked if I was writing a second volume. Almost every researcher I’ve talked to [says] the book has resonated with [them], and everyone has their own stories.
David Streiner is the author of nine books, most recently A Guide for the Statistically Perplexed: Selected Readings for Clinical Researchers (2013). He has been a professor in the department of clinical epidemiology & biostatistics at McMaster University for 30 years, a professor in the department of psychiatry at the University of Toronto for 10 years, and founding director of the Kunin-Lunenfeld Applied Research Unit at the Baycrest Centre for Geriatric Care.
Souraya Sidani, his co-editor of When Research Goes Off the Rails: Why It Happens and What You Can Do About It, is professor and Canada Research Chair at the School of Nursing at Ryerson University.
This interview has been edited and condensed.