A Classification Framework for Web Robots
Document Type
Article
Publication Date
12-2012
Abstract
The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year-long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.
Repository Citation
Doran, D.,
& Gokhale, S. S.
(2012). A Classification Framework for Web Robots. Journal of the American Society for Information Science and Technology, 63 (12), 2549-2554.
https://corescholar.libraries.wright.edu/cse/253
DOI
10.1002/asi.22741
