Bibliometric indicators - use with caution
by Peggy Berkowitz
Citation counts and other bibliometric indicators should be used with "great caution" in the social sciences and humanities, because researchers in those disciplines publish far fewer journal articles than researchers in the natural and medical sciences says a new study.
The study by Science-Metrix for the Social Sciences and Humanities Research Council shows that journal articles represent just 45 to 70 percent of publications in the social sciences and just 20 to 35 percent of publications in the humanities.
"Especially in several fields in the humanities, journal articles weren't important at all," says Eric Archambault, the study's author and president of Science-Metrix. "If you're going to measure journal citations, you have to know you're measuring just 10 percent of [scholarly] output. The other 90 percent is in books and conference proceedings." Only economics really lends itself to bibliometric analysis, he adds.
Although the findings aren't surprising, the study is important because it's the largest and most comprehensive of its kind for the humanities and social sciences, says Dr. Archambault. SSHRC says the study is already being requested by researchers in Canada and other countries.
Bibliometrics creates statistics by examining bibliographies in a body of literature to reveal patterns of authorship, publication and use. One of the most common indicators is the "citation index," created by counting the number of times particular articles are cited by other scholarly writers. The index is often used to evaluate the relative importance of various papers, researchers and countries.
The study also underlined another problem: bibliometric analyses over-rank English-speaking countries and English-language publications because they are often based on English-only databases. This is not such a big problem in the natural sciences, says Dr. Archambault, where English is "the clear language of publication." But in the social sciences and humanities, local variables and phenomena can be extremely relevant, so researchers may prefer to publish their findings in local journals in their own language.
"When [the indices] rank countries, the English-speaking countries are all at the top," notes Dr. Archambault, who teaches science metrics at Université du Québec à Montréal. "For me it's a bit bizarre to see Australia beating France and Germany in the social sciences," when both France and Germany have a longer history of publishing in social sciences.
Dr. Archambault says bibliometric analyses are still useful for mapping exercises - to track networks of collaboration, for example.
In fact, SSHRC commissioned a bibliometric analysis of its own as one of the studies for its self-transformation exercise. It asked the Observatoire des sciences et des technologies to map trends in collaboration among Canadian researchers in the humanities and social sciences between 1980 and 2002.
This study found that multi-authored articles in the social sciences more than doubled, from 30 percent of all articles to nearly 70 percent. In the humanities, just 10 percent of articles were by more than one author, remaining stable during the 22-year period. By comparison, 90 percent of articles in the natural sciences and engineering were written by more than one author.
When SSHRC-discipline authors collaborated, more than half the time they did so with scholars from the United States. Less than one percent of publications involved authors from two or more provinces.
Other research commissioned by SSHRC included a roundtable with 16 scholars in the early stages of their careers and a study on how humanities research is disseminated in the mass media. Short summaries of most of the commissioned studies are available on SSHRC's Web site and researchers may request copies of the studies from SSHRC.
