Ranking and Relevance in Noormags and RICEST Databases

The main purpose of information retrieval systems is to retrieve relevant information for users. This means that the results of the search must answer the questions provided to the system. Therefore, the evaluation of relevance is very important in such systems. In addition to relevance, the order and placement of articles are also important to the user. The retrieval systems should put more relevant articles at the top of the retrieval list. Evaluating the quality of ranking performance is a key activity in the field of information retrieval. This article assesses relevance and ranking of two databases.

The sample includes 390 Persian articles retrieved in each of the Noormags and RICeST databases. For each topic inquired were carried out in both databases in two phases within the span of one month. The first 10 articles retrieved from each database were recorded based on the system ranking. Relevance score was given by 3 subject specialists within the range of zero to ten. Spearman correlation test was used to compare the ranking of the system with the ranking of the user. Data analysis was performed using descriptive and inferential statistics using SPSS software. The distance precision formula carried out to check the accuracy of the retrieval precision of related documents in the two databases, and the expected Reciprocal Rank was used to evaluate the quality of the ranking of articles.

Users were far less familiar with RICeST database. Significant, consistent, and moderate correlation was found between system rankings and user rankings at the Noormags database in the first phase, i. e., ranking by users increases or decreases as the system rank increases or decreases. We found significant, consistent, and strong correlation between system’s ranking and user ranking in Noormags in the second phase. However, there was no correlation between system ranking and user ranking in RICeST database in both the first and second phases. Therefore, Noormags database ranking was found closer to the users’ ranking. Ranking quality by Noormags was relatively better than that of RICeST. Also, accuracy of the relevance precision of Noormags articles was higher than RICeST. From the users' point of view, Noormags database retrieved more relevant documents.

Noormags' new algorithms and capabilities have increased the relevance and ranking of its output. The findings could help database administrators to upgrade their databases by taking advantage of technologies to make semantic retrieval possible.