CleanupSpider 

 

MakeNetDate 

 

CacheFolder 

Location for temporary files saved by the spider.  

LinkTraceFilename 

Filename for link trace file recording each link found by the spider and how it was processed.  

Add 

Add a web site to the list of sites to crawl  

GetNextDoc 

Get the next document from the data source.  

GetStatus 

Get the status of the spider  

IsCrawling 

Returns true if the spider is still downloading data  

Rewind 

Initializes the spider so the next GetNextDoc call will return the first document.  

SetDebugLogging 

Create a debug log recording the spider's actions  

StartCrawl 

Starts crawling web sites on a different thread.  

StopCrawl 

Requests that the spider stop crawling web sites