Spider Options

Menu option: Options > Preferences > Spider options

Automatically log on to web sites on my local area network
When you index web sites on your local area network, dtSearch can attempt to log on to the sites using your Windows username and password.  Un-check this box if you would prefer not to use your Windows username and password to log on this way.

Logon even if a site only supports non-secure authentication methods
Some web sites only support "Basic" authentication, a type of authentication that requires your password to be sent across the internet without encryption.  Un-check this box to prevent dtSearch from logging on to a site that does not support secure authentication methods.

Do not prompt for a password if access to a site is denied
If the dtSearch Spider receives an "Access denied" response from a web site when it tries to download a page, and if no password is found for the site in the web site options, then the Spider will prompt for a user name and password to access the page.   Check this box to prevent password prompts so that the Spider will continue without interruption.

Use Internet Explorer to download web pages
The dtSearch Spider will use the WinHTTP library to download web pages, unless this box is checked.  Use this option if you want the dtSearch Spider to use your Internet Explorer browser settings to access the internet (for example, to use a proxy server).

Log the links found on each page in spiderlog.txt
To determine how the Spider found a particular page or file on a web site, check this box before indexing the site.  After the index is built, the file spiderlog.txt in the index folder will contain a list of every link found on every page indexed.  

Folder to use for temporary files
By default, dtSearch will use a sub-folder under your Windows "TEMP" folder for temporary files downloaded by the Spider.  You can specify a different location here if there is not enough space on the drive where your TEMP folder is located.

Timeout limit for downloading pages
This is the maximum amount of time that you want the Spider to wait for a web page to download before giving up and moving on to the next page.