Location for temporary files saved by the spider.

Filename for link trace file recording each link found by the spider and how it was processed.

Add a web site to the list of sites to crawl

Get the next document from the data source.

Get the status of the spider

Returns true if the spider is still downloading data

Initializes the spider so the next GetNextDoc call will return the first document.

Create a debug log recording the spider's actions

Starts crawling web sites on a different thread.

Requests that the spider stop crawling web sites