Efficient Document Partitioning for Load Balancing between Servers Using Term Frequency of Past Queries

The main goal of web search engines is to find the most relevant results with respect to the user query in a shortest possible time. To do so, the crawled documents have to be partitioned between several servers in order to use their aggregate retrieval and processing power. The search engines use different policies for efficient partitioning of documents. In this paper, we propose a new document partitioning method that intends to balance the load between servers to reduce the response time of queries. The idea is to weigh each term based on its daily frequency in log of past queries. We then assign a weight to each document via summing the weight of its substituent terms. The weight of a document approximates the likelihood of its presence in future search results. Finally, the documents are partitioned between servers in a way that the sum of document weights in each server becomes roughly equal. Our evaluation results show that the proposed method is able to balance the load by about 20% better than former algorithms, especially in the peak of search engine traffic.