A Correlation Study of Bibliometric-Based and Informed-Peer-Review University Rankings: The Case of UK Research Excellence Framework (REF) and the World's Prestigious University Ranking Systems

Informed peer-review university ranking systems like REF involve peer review to achieve high-quality university performance evaluations while considering bibliometric facts. Despite its advantages and methodological success, it is not possible to implement the review-based exercise in all countries due to unaffordable costs, vulnerability to bias, and differences in cultural, economic, managerial, and infrastructural conditions. As an alternative, some science systems prefer to consider using the results of international university rankings to gauge the academic performance of their higher education institutions. Using a different methodology, the international ranking systems tend to maintain their efficiency by heavily relying on the bibliometric information extracted from databases, the performance data gathered from official authorities, and academic and employer surveys. This gives rise to the question of to what extent these two types of systems are convergent in their results. If their different methodologies lead to similar results, they could be interpreted to have similar effectiveness in their evaluation of academic performance. Therefore, by relying on the results of the international ranking systems, one can avoid the shortcomings of the review-based method, and maintain both the efficiency and effectiveness of the evaluation systems. To reveal the convergence of the results obtained from the methods, the present study explores a sample of British universities evaluated by REF (2014) to investigate the correlation between their scores in REF and the world’s prestigious university rankings.

Using a quantitative content analysis method, the present study concentrates on a collection of 150 British universities evaluated simultaneously by REF (2014) and at least one of the world's prestigious ranking systems including QS, THE, Leiden and ARWU. Due to the small size of the population, all REF members are examined without sampling. The evaluation results of these universities are extracted from these systems and entered into a checklist. Also, the subject fields and disciplines covered by the universities are collected to investigate their probable effects on the results. The universities’ subject coverage similarity is calculated using the Cosine similarity measure and K Nearest Neighbor technique in the KNIME data mining platform. Finally, correlation and regression analyses are applied to analyze the data.

UK universities’ scores in REF are found to be significantly correlated to theirs in international ranking systems. They are highly correlated to QS and THE’s, while being moderately associated with ARWU’s, and weakly-to-strongly correlated to Leiden’s. The regression analyses show no significant effects of subject coverage on the overall scores, except for medical tendency’s effect on QS. However, the subject coverage affects some dimension scores. While it does not significantly predict any of the dimensions of ARWU, it partially predicts the Citations and Industry Income dimensions in THE, with medical and technical, and engineering subjects respectively having the highest positive predicting power. In QS, the subject coverage partially predicts the dimensions of Academic reputation, Faculty/student ratio, International faculty ratio, International student ratio, and Citation Per faculty. Medical science has the highest positive effect on the dimensions of Academic reputation, Faculty/student ratio, and Citations per faculty. Moreover, basic sciences have the highest negative effect on the International student ratio and International faculty ratio. Also, the subject coverage can predict Leiden scores for the dimensions PP (top 10%) (i.e. the proportion of a university’s publications that, compared with other publications in the same field and the same year, belong to the top 10% most frequently cited), MCS (i.e. the average number of citations of the publications of a university), MNCS (i.e. the average number of citations of the publications of a university, normalized for field and publication year), PP(collab) (i.e. the proportion of a university’s publications that have been co-authored with one or more other organizations), PP(int collab) (i.e. the proportion of a university’s publications that have been co-authored by two or more countries), PP(industry) (i.e. the proportion of a university’s publications that have been co-authored with one or more industrial organizations) and PP(>1000 km) (i.e. the proportion of a university’s publications with a geographical collaboration distance of more than 1000 km). Medical science has the highest positive predictive power for the scores in all the mentioned dimensions except for PP(industry).

The results of REF, as a peer-review-based university ranking informed with bibliometric data, are highly correlated to those of the international evaluations based on performance statistics and enriched by surveys, while being moderately correlated to those performed by just performance statistics. The subject coverage impact on the rankings challenges the application of the results in comparing universities with different subject coverages.