Long Range Dependence (LRD) in the Arrival Process of Web Robots

Document Type

Conference Proceeding

Publication Date


Find this in a Library

Catalog Record


There is strong evidence to suggest that a significant proportion of traffic on Web servers, across many domains, can be attributed to Web robots. With the advent of the Social Web, widespread use of semantic Web technologies, and development of service-oriented Web applications, it is expected that this proportion will only rise over time. One of the most important distinctions between robots and humans is the pattern with which they request resources from a Web server. In this paper, we examine the arrival process of Web robot requests across Web servers from three diverse domains. We find that, regardless of the domain, Web robot traffic exhibits long range dependence (LRD) similar to human traffic. We discuss why, at least in some cases, LRD in robot traffic may not be generated by heavy-tailed response sizes as in the case of human traffic.


Presented at the International Conference on Intelligent Network and Computing, New Delhi, India, August 18-19, 2012.

Catalog Record