Description

CrawlDelay is the default time to wait between requests to a given host. If a specific delay is specified in the relevant robots.txt, then this delay is used instead. Crawl delay can be customized further by implementing the ComputeDelay extender function.

CrawlDelay is referenced in 9 repositories

github.com/PuerkitoBio/gocrawl github.com/barthr/crawler-broker github.com/iScr/golangCrawl

...