Description

NewCrawler returns a Crawler initialized with the default Options' values and the provided Extender. It is highly recommended to set at least the Options.RobotUserAgent to the custom name of your crawler before using the returned Crawler. Refer to the Options type documentation for details.

NewCrawler is referenced in 1 repository

github.com/PuerkitoBio/gocrawl