A Classification Framework for Web Robots
The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year-long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.
& Gokhale, S. S.
(2012). A Classification Framework for Web Robots. Journal of the American Society for Information Science and Technology, 63 (12), 2549-2554.