View definition


Defined in


RobotUserAgent is the user-agent value of the robot, used to find a matching policy in the robots.txt file of a host. It is not used to make the robots.txt request, only to match a policy. It should always be set to the name of your crawler application so that site owners can configure the robots.txt accordingly.

RobotUserAgent is referenced in 1 repository