Tracking Scholarship in the Arts & Humanities

Purpose – The purpose of this paper is to make an explicit case for the use of data with contextual information as evidence in arts and humanities research evaluations rather than systematic metrics. Design/methodology/approach – A survey of the strengths and limitations of citation-based indicators is combined with evidence about existing uses of wider impact data in the arts and humanities, with particular reference to the 2014 UK Research Excellence Framework. Findings – Data are already used as impact evidence in the arts and humanities but this practice should become more widespread. Practical implications – Arts and humanities researchers should be encouraged to think creatively about the kinds of data that they may be able to generate in support of the value of their research and should not rely upon standardised metrics. Originality/value – This paper combines practices emerging in the arts and humanities with research evaluation from a scientometric perspective to generate new recommendations.

The well-documented limitations of journal impact factor rankings and perceptual ratings, the evolving scholarly communication system, the open-access movement, and increasing globalization are some reasons that prompted an examination of journal value rather than just impact. Using a single, specialized journal established in 1960, about education for the Information professions, the author discusses the fall from citation grace of the Journal of Education for Library and Information Science (JELIS) in terms of impact factor and declining subscriptions. Journal evaluation studies in Library and Information Science based on subjective ratings are used to show the high rank of JELIS during the same period (1984–2004) and explain why impact factors and perceptual ratings either singly or jointly are inadequate measures for understanding the value of specialized, scholarly journals such as JELIS. This case study was also a search for bibliometric measures of journal value. Three measures, namely journal attraction power, author associativity, and journal consumption power, were selected; two of them were redefined as journal measures of affinity (the proportion of foreign authors), associativity (the amount of collaboration), and calculated as objective indicators of journal value. The affinity and associativity for JELIS calculated for 1984, 1994, 2004, and consumption calculated for 1985 and 1994 show a holding pattern; however, they also reveal interesting dimensions for future study. Journal value is multidimensional and citations do not capture all the facets; costs, benefits, and measures for informative and scientific value must be distinguished and developed in a fuller model of journal value.

Bibliometric evaluations of research outputs in the social sciences and humanities are challenging due to limitations associated with Web of Science data; however, background literature has shown that scholars are interested in stimulating improvements. We give special attention to book reviews processed by Web of Science history and literature journals, focusing on two types: Type I (i.e., reference to book only) and Type II (i.e., reference to book and other scholarly sources). Bibliometric data are collected and analyzed for a large set of reviews (1981–2009) to observe general publication patterns and patterns of citedness and co-citedness with books under review. Results show that reviews giving reference only to the book (Type I) are published more frequently while reviews referencing the book and other works (Type II) are more likely to be cited. The referencing culture of the humanities makes it difficult to understand patterns of co-citedness between books and review articles without further in-depth content analyses. Overall, citation counts to book reviews are typically low, but our data showed that they are scholarly and do play a role in the scholarly communication system. In the disciplines of history and literature, where book reviews are prominent, counting the number and type of reviews that a scholar produces throughout his/her career is a positive step forward in research evaluations. We propose a new set of journal quality indicators for the purpose of monitoring their scholarly influence.

The severity of the global economic crisis has put the spotlight firmly on measuring academic and research performance and productivity and assessing its contribution, value, impact and benefit. While, traditionally, research output and impact were measured by peer-publications and citations, there is increased emphasis on a ‘market-driven approach’, which favours the bio-, medical and technological sciences, and has helped reinforce a disciplinary hierarchy in which arts and humanities research has struggled for attention. This article charts the changing policy environment across Ireland, the Netherlands and Norway. It draws on evidence from the HERAVALUE project that studied how different stakeholders value arts and humanities research; almost 100 interviews were conducted with representatives from the academy, policymakers and civil society in these three countries. Although the arts and culture have played a distinctive nation-forming role, and continue to do so, each country has adopted very different policy responses towards arts and humanities research.

Understanding the value of the arts to individuals, families and communities has long been a query for policy-makers and researchers. At the core of the value system is how individuals are affected and transformed by virtue of participating. Individual arts experiences are the building blocks upon which the larger outcomes of the arts, such as social cohesion and cognitive development, are constructed. Tools for effectively measuring how art affects audiences, however, are just being explored. Anecdotal stories of impact provide helpful information. But as arts groups are called upon to provide more rigorous evidence of the impacts of their artistic programmes, a more systematic measurement approach is needed to convey the intrinsic value of their work. This article reviews initial exploratory research and subsequent research undertaken over the past six years to better understand how art affects audiences, and to develop reliable and replicable measurement approaches. The research has yielded a more nuanced, but still emergent, understanding of the emotional, intellectual, social and aesthetic constructs of intrinsic impact, methods of measuring them, and insight into the limitations and applications of this work for policy-makers and arts practitioners.

The importance of emphasizing the societal impact of scientific research and how such research benefits society are addressed. New research into measures that can assess the societal impact of research would provide better qualitative and quantitative data on which funding agencies and politicians could base decisions. At the same time, such measurement should not come at the expense of basic, blue-sky research, given that it is and will remain near-impossible to predict the impact of certain research projects years or decades down the line.

This article studies citation practices in the arts and humanities from a theoretical and conceptual viewpoint, drawing on studies from fields like linguistics, history, library & information science, and the sociology of science. The use of references in the humanities is discussed in connection with the growing interest in the possibilities of applying citation analysis to humanistic disciplines. The study shows how the use of references within the humanities is connected to concepts of originality, to intellectual organization, and to searching and writing. Finally, it is acknowledged that the use of references is connected to stylistic, epistemological, and organizational differences, and these differences must be taken into account when applying citation analysis to humanistic disciplines.

The introduction of ‘impact’ as an element of assessment constitutes a major change in the construction of research evaluation systems. While various protocols of impact evaluation exist, the most articulated one was implemented as part of the British Research Excellence Framework (REF). This paper investigates the nature and consequences of the rise of ‘research impact’ as an element of academic evaluation from the perspective of discourse. Drawing from linguistic pragmatics and Foucauldian discourse analysis, the study discusses shifts related to the so-called Impact Agenda on four stages, in chronological order: (1) the ‘problematization’ of the notion of ‘impact’, (2) the establishment of an ‘impact infrastructure’, (3) the consolidation of a new genre of writing–impact case study, and (4) academics’ positioning practices towards the notion of ‘impact’, theorized here as the triggering of new practices of ‘subjectivation’ of the academic self. The description of the basic functioning of the ‘discourse of impact’ is based on the analysis of two corpora: case studies submitted by a selected group of academics (linguists) to REF2014 (no = 78) and interviews (n = 25) with their authors. Linguistic pragmatics is particularly useful in analyzing linguistic aspects of the data, while Foucault’s theory helps draw together findings from two datasets in a broader analysis based on a governmentality framework. This approach allows for more general conclusions on the practices of governing (academic) subjects within evaluation contexts.