Crawl observation lab

This independent test site records when web crawlers discover a newly published URL. Each experiment uses a fresh address to distinguish a requested fetch from a revisit.

What is measured

The server records the request time, HTTP response, source address, and browser identification. A mobile crawler must identify an Android device and pass independent reverse and forward DNS checks. A copied user agent is not sufficient.

Limits

A request demonstrates retrieval of a page, not inclusion in a search index. Discovery, crawling, rendering, and indexing are separate events.