Discovering New Trends in Web Robot Traffic through Functional Classification
This paper proposes a novel functional classification scheme to understand and analyze web robot traffic. The scheme is rooted in the recognition that the crawling behavior of a robot on a site is primarily governed by its intended purpose or functionality. We apply the classification rules to analyze web server access logs from the University of Connecticut School of Engineering domain. The analysis results indicate how the classification scheme can provide insights into the robot traffic based on their functionality.
& Gokhale, S. S.
(2008). Discovering New Trends in Web Robot Traffic through Functional Classification. Proceedings of the Seventh IEEE International Symposium on Network Computing and Applications, 275-278.