View definition

(*DefaultExtender).RequestRobots

Defined in github.com/PuerkitoBio/gocrawl/ext.go

Description

RequestRobots asks the worker to actually request (fetch) the robots.txt.

RequestRobots is referenced in 0 repositories