Rethinking the Recall Measure in Appraising Information Retrieval Systems and Providing a New Measure by Using Persian Search Engines

The aim of the study was to improve Persian search engines’ retrieval performance by using the new measure. In this regard, consulting three experts from the Department of Knowledge and Information Science (KIS) at Ferdowsi University of Mashhad, 192 FUM students of different degrees from different fields of study, both male and female, were asked to conduct the search based on 32 simulated work tasks (SWT) on the selected search engines and report the results by citing the related URLs. The Findings indicated that to measure recall, one does not focus on how documents are selecting, but the retrieval of related documents that are indexed in the information retrieval system database is considered While to measure comprehensiveness, in addition to considering the related documents' retrieval in the system's database, the performance of the documents selecting on the web (performance of crawler) was also measured. At the practical level, there was no strong correlation between the two measures (recall and comprehensiveness) however, these two measure different features. Also, the test of repeated measures design showed that with the change of the measure from recall to comprehensiveness, the search engine’s performance score is varied. Finally, it can be said, if the study purpose of the search engines evaluation is to assess the indexer program performance, the recall use will be suggested while, if its purpose is to appraise the search engines to determine which one retrieves the most relevant documents, the comprehensiveness use will be proposed.