WebSite.IgnoreRobotsTxt Field

If true, the Spider will crawl areas of the site even if robots.txt excludes them.

public bool IgnoreRobotsTxt;