How to exclude folders from an index

Article: dts0135

When you specify a folder to be indexed in dtSearch, by default all subfolders of that folder will also be indexed. You can change this and have dtSearch index just the top-level folder, but sometimes you may want to exclude a few folders from a large tree of folders and subfolders.

To do this, use the Exclude filters in the Update Index dialog box. In addition to the usual type of filters -- *.exe, *.tif, etc. -- you can include filters with directory information. For example, to exclude everything in the c:\docs\archive\oldfiles folder, you could add an exclude filter "c:\docs\archive\oldfiles\*"

If a filter contains a space, it must be quoted, like this:  "*\name with space\*".

Exclude filters will apply inside a ZIP archive.  For example, if you exclude *.doc from an index, and a ZIP archive contains sample.doc, then the indexer will skip sample.doc because of the exclude filter, even though it is inside a ZIP.

Additional Information

When a filename filter includes a \, dtSearch will compare it to the full path and filename of a document, but if the filename filter does not include a \, dtSearch will compare it to the filename alone. Examples:

 

s*.doc

Matches c:\docs\sample.doc, but not s:\docs\other.doc

*\oldfiles\*

Matches any file in a folder named oldfiles

"*\Old Files\*"

Matches any file in a folder named "Old Files".  The quotation marks are needed because "Old Files" has a space, and without the quotation marks, dtSearch will see the filter as two separate filters, *\Old and Files\*

Web Sites and the dtSearch Spider

You can use the same type of filters to control which pages of a web site the dtSearch Spider will visit.   For example, if you have a filename filter "*/OnlyThisFolder/*", then only documents under the "OnlyThisFolder" folder will be indexed.

Additionally, the dtSearch Spider obeys the instructions in a robots.txt file if one is present on the web site, and it also obeys instructions in a robots META tag if one is present in an HTML file.  This makes it possible to create a file with links in it that dtSearch will follow, while preventing dtSearch from indexing the links.   For more information on robots.txt and robots META tags, see:

https://www.robotstxt.org