Authentication 

Authentication settings to use to connect to a web site.  

CrawlDepth 

Number of links from the start page to follow.  

ExcludeFilters 

Filename filters indicating which pages should be not indexed.  

FormAuthentication 

Form authentication settings to use to connect to a web site that uses HTTP GET or POST requests for authentication.  

IgnoreRobotsTxt 

If true, the Spider will crawl areas of the site even if robots.txt excludes them.  

IncludeFilters 

Filename filters indicating which pages should be indexed.  

MaxItemsToIndex 

Use this setting to limit the number of pages the Spider should index on this web site.  

MaxSizeToIndex 

Use this setting to limit the maximum size of files that the Spider will attempt to access.  

PageTimeoutSeconds 

Number of seconds to wait before timing out when trying to download a single page.  

Proxy 

Proxy settings to use to connect to a web site.  

ServerFilters 

List of server names other than the starting server that the Spider can visit.  

SiteTimeoutSeconds 

Use this setting to limit the amount of time the Spider will spend crawling pages on this web site.  

Url 

Starting URL for the crawl.  

UserAgent 

Name to use to identify this program to the web server  

WaitBetweenPagesMillis 

Number of milliseconds to wait between page downloads.  

Copy