Description

NewOptions creates a new set of Options with default values using the provided Extender. The RobotUserAgent option should be set to the name of your crawler, it is used to find the matching entry in the robots.txt file.

NewOptions is referenced in 9 repositories

github.com/PuerkitoBio/gocrawl github.com/herenow/sandy-crawler github.com/barthr/crawler-broker

...