What are the alternatives to university rankings in assessing research? – opinion

Global efforts to find new ways to assess research quality have been underway for more than 10 years now, with the Declaration on Research Assessment (DORA) celebrating its 10^th anniversary in May 2023.

This initiative and other related initiatives such as the Leiden Manifesto and the Hong Kong Principles have sought to show that there are better ways of assessing research and researchers than current ranking systems, which are often commercially owned. 

DORA also highlights that, regardless of who makes the measurements, we need assessments of research publication quality that are sensitive to discipline differences and which support different research goals and objectives.

Many would argue that university rankings are equally inappropriate in assessing diverse missions of universities and their broader purposes beyond research.

The arguments would be that they do so for the same underlying reason of suiting the purposes of commercial gain by ranking companies and publishers. 

What about the matters that are important to the experiences of staff and students and university partners? How many rankings measure staff culture and feelings of safety on campus?  How many rankings genuinely measure and include what we all say is important, that we value the student experience?

Research value, and its importance to our society, arises in part from its diversity in focus and approach and its ability to address a broad range of questions.

If we subject research value to narrow and prescribed ways of measuring and assessing it, we are in danger of restricting breadth of enquiry, discriminating between approaches to pursuing it, and introducing motives for gaming of the system.

Some of the biggest challenges to research exploration have come from systems restricting inter-disciplinarity and assessing impact.

Early journals that celebrated inter-disciplinarity and open access, such as PLOS ONE, had to overcome the tendency of journal impact factors and research grant awarding bodies to value established disciplines and outlets more highly.

The most recent university rankings from Times Higher Education (THE) have described themselves as focussing on impact, with the most recent 2023 awards seeing some Australian universities achieve high rankings. 

Inevitably, these rankings are celebrated in how universities promote themselves to audiences.

The measurements are made using a methodology created by THE, within the wider framework of the UN Sustainable Development Goals. 

Other universities, of the many that did not enter the rankings, took the opportunity to publicly launch sustainability reports to demonstrate their own commitment to sustainability in terms of how they measure it, rather than THE Impact rankings. This included the University of Melbourne.

Other changes are happening in research assessment – for example, how we think about curricula vitae (CVs) is changing from current practices of more quantitative type CVs (i.e. listing every publication and grant without identifying themes and main contributions) to the concept of narrative CVs.

Narrative CVs differ by allowing significant and themed outcomes and outputs to be discussed within a more individualised narrative of impact, with a smaller number of papers listed.

Are there lessons here of how different universities might choose how they want to describe their impact in regard to their purpose and goals? They could do so by choosing to demonstrate their impact meaningfully rather than be subjected to ubiquitous and universal one size fits all rankings.

The problem with taking a one size fits all approach is also that it discriminates in terms of variability in opportunity and intent.

It results in encouraging unduly rule-bound approaches to measuring and promoting research. And it creates incentives to game the system.

DORA, as a truly global initiative, aims to ensure that research assessment reform is relevant to both developed and developing countries.

Furthermore, within nations, it aims to understand and reflect the relative mission and opportunity between large and established metropolitan universities and smaller and newer regional universities that serve vitally important local and community employment and engagement opportunities. 

We have been pioneers in Australia in some of our research funding schemes in recognising outcomes relative to opportunity (ROPE).

We have seen important initiatives in both the NHMRC and ARC, though many of these initiatives between bodies are not consistent.

A number of these ROPE approaches have since found their way into institutional promotion schemes and research management systems, and provide opportunities for better assessment of publications, rankings, grant awards, and institutional reward and recognition schemes.

The ultimate route for meaningful research assessment is for researchers themselves – and universities and grant-awarding bodies – to take ownership of the issues, to counter discrimination, and to focus on genuine research purpose and impact, over the gains to be had from playing the game.

This generates an enormous opportunity but also a clear need for leadership within the research sector.

After 10 years of DORA it is now clear that there is a global appetite for reform in research assessment, especially to ensure that research is assessed with integrity.

Ginny Barbour is vice-chair of the Declaration on Research Assessment (DORA)

Martin Betts, co-founder of HEDx.

Do you have an idea for a story?
Email [email protected]