Web Robot Detection Using Fuzzy Rough Set Theory

Web robots are software programs that traverse the internet autonomously. Their most important task is to fetch information and send it to the origin server. The high consumption of network bandwidth by them and server performance reduction, have caused the web robot detection problem. In this paper, fuzzy rough set theory has been used for web robot detection. The proposed method includes 4 phases. In the first phase, user sessions have identified using fuzzy rough set clustering. In the second phase, a vector of 10 features is extracted for each session. In the third phase, the identified sessions are labeled using a heuristic method. In the fourth phase, these labels are improved using fuzzy rough set classification. The proposed method performance has been evaluated on a real world dataset. The experimental results have been compared with state-of-the-art methods, and show the superiority of the proposed method in terms of F-measure.