See Also
WebSite Data Members | WebSite Structure | WebSite Members | dtSearch::Spider Namespace
dtSearch Text Retrieval Engine -- .NET API (.NET 2.0-4.0) 7.70
Contents | Index | Home
Previous | Up | Next
WebSite::IgnoreRobotsTxt Data Member

If true, the Spider will crawl areas of the site even if robots.txt excludes them.

C#
public bool IgnoreRobotsTxt;
Visual Basic
Public IgnoreRobotsTxt As bool
See Also
WebSite Data Members | WebSite Structure | WebSite Members | dtSearch::Spider Namespace
You are here: dtSearch::Spider Namespace > WebSite Structure > WebSite Data Members > IgnoreRobotsTxt Data Member
Copyright (c) 1998-2012 dtSearch Corp. All rights reserved.