Imagine you’re a deputy minister of health and you wished to take a look at what data is being collected on the healthcare system. You discover that you know only a little about the problems patients come in with, that you stopped collecting data on doctors years ago, and that you have almost no data on the overall health of the country. How can you possibly make decisions?
This is not the actual situation in Canada for health care, but a situation analogous to this does exist for postsecondary education, noted Glen Jones at a recent panel discussion on “PSE data gaps” held June 2 during the annual conference of the Canadian Society for Studies in Higher Education, one of the 70 associations that took part in the Congress of the Humanities and Social Sciences at the University of Ottawa.
I served as the moderator for the panel. I was asked to participate because of recent blog posts I had written about the lack of good data on Canada’s PSE system.
Dr. Jones, a well-known expert and professor of higher education at the Ontario Institute for Studies in Education of the University of Toronto, said we’ve been in this situation for years: “We have some basic data on students. We have some data on a variety of usually one-off longitudinal sets. We decided as a nation to stop collecting data on faculty, although there has been some attempt to compensate for that at the institutional level. But the reality is that the amount of data available within the [PSE] system is remarkably small.”
When he does international comparative work and looks at what data are available in other national systems, said Dr. Jones, “Canada stands out as being quite remiss.” If you talk to anybody at the OECD behind closed doors, he added, “one of the first things they’ll ask you is what’s going on with data in Canada? It’s just embarrassing.”
Dr. Jones said data collection should be thought of as a piece of important public infrastructure. He said this kind of data is central to evidence-based policy discussions inside government. “But perhaps just as importantly, it is important in terms of our work as a nation, to come to grips with some key policy challenges and dimensions within higher education.” He added: “A lot of the conversations we have today about issues of skills and the relationship between higher education and the labour market are vacuous in large part because we have so little data to link these issues together.”
To solve the problem, he suggested that a much stronger data system is needed, one that is not restricted to Statistics Canada. While there have been attempts to combine data at the university level and to share data among institutions, there are limitations “because there are things universities don’t really want to count, or feel uncomfortable counting or are expensive to count.” There is not, acknowledged Dr. Jones, “a lot of political advantage spending money on data collection.”
Next up on the panel was Alexandre Beaupré-Lavallée, a junior professor in higher education and educational administration at the school of educational sciences at Université de Montréal. From a researcher’s perspective, he said, the problem is not just missing data, but the quality of the data that does exist and access to it.
One problem in terms of quality, he said, is that data collection is often voluntary, and therefore institutions may opt out from filling in particular data fields. The data on sessional instructors is an example of that, as some institutions may be concerned about what the data will show or how they’ll be interpreted.
As for access, Dr. Beaupré-Lavallée mentioned the huge amount of what he calls “dark literature” on higher education. “The sheer magnitude of data produced by institutions is amazing … but for strategic reasons usually this literature is not shared, is not publicized even though most of these data are from public institutions.” Access to that data, he said, could probably fill a lot of the gaps. Dr. Beaupré-Lavallée also would like to see a centralized repository for PSE data. “I’m a faculty member in higher education, and I have a hard time keeping track of everything that’s been done.”
It was this latter point that prompted Michelle Nilson, another of the panelists, to recently create the Compendium of Canadian Postsecondary Education Data Sources. Dr. Nilson, an assistant professor in the faculty of education at Simon Fraser University, said that when she came to Canada in 2006, she immediately realized that, unlike in the U.S., there was “no central place for me and my students to go in terms of getting the data.”
Dr. Nilson calls the project “a living document,” adding, “It’s not by any means complete … but I’m hoping it will push the conversation forward.”
The compendium groups the data into various categories, such as “Students and Learning,” “Faculty and Staff,” “Finances and Facilities,” and so on. There are three inclusion criteria for the compendium: the data must be Canadian, at any level (provincial, federal, institutional, etc.); they must be uploaded or updated periodically (i.e., not one-off data); and they must be primary data sources. Each listing includes the organization, dates, description of the resource, the data collected, key words, and usually a link to the data set, along with any comments. Her graduate students are working on the compilation project.
The fourth and final panel member had a very different message for the audience. Ross Finnie, a professor in the graduate school of public and international affairs at the University of Ottawa, said that while he agreed with most of what had been said, “on the other hand I look for the opportunities and I think the opportunities are greater than they’ve ever been before, we just need to push it over the top.”
Longitudinal surveys, like the National Graduates Survey or the now defunct Youth in Transition Survey, will always have their role, he said. “But, the real opportunities going forward are with administrative data, and that’s what I have plunged into.” Every institution, he said, collects huge amounts of data. For example, “they know so much about the student at the point of entry, their high-school grades, every course they take, how they did in every course – there’s an incredible richness of information.”
The rich data allows researchers to follow every student as they move through their studies, and then link that with tax data post-graduation to learn about employment of graduates from different programs and socioeconomic classes and many other outcomes.
Dr. Finnie did a pilot project with the University of Ottawa and had 98 percent of graduates linked to the tax data and then followed their graduation outcomes for up to 13 years. His research team is scaling up the research and now has 12 institutions participating. He’d like to see more participating, but the provinces have been unwilling to compel institutions to take part.
The one counter example, said Dr. Finnie, is British Columbia. The province collects all the administrative data on PSE and tracks students as they go through the system, making it easier for Dr. Finnie to track labour market outcomes and link these back all the way to K-12. “There are no data like that in the world. We’re sitting on those data, we can push them further, but what we need is collaboration. We need everyone to buy in.”