View definition

(*URLContext).IsRobotsURL

Defined in github.com/PuerkitoBio/gocrawl/urlcontext.go

Description

IsRobotsURL indicates if the URL is a robots.txt URL.

IsRobotsURL is referenced in 1 repository

github.com/PuerkitoBio/gocrawl