Multithreaded operations

Last Reviewed: April 6, 2014

Article: DTS0233

Applies to: dtSearch Engine 6, 7

The dtSearch Engine supports multithreaded indexing, searching, and file conversion. As a general rule, any API function can be called from any thread at any time. However, access to API objects such as SearchResults, IndexJob, FileConvertJob, and SearchResults is not synchronized, so a single API object should not be accessed from more than one thread at a time. 

For convenience, the IndexJob and SearchJob objects have ExecuteInThread methods that will start an indexing or searching job in a separate thread and return immediately. You can then call IsThreadDone or AbortThread periodically to check the status of a thread that has been started.  Alternatively, your application can handle creation of threads directly and can invoke IndexJob, SearchJob, and other API objects from different threads as needed.

Concurrent index access

Any number of SearchJobs can target the same index, or different indexes, concurrently in different threads or in different processes.  Searching is done without any need for file or record locks, so aside from the need to share CPU and other hardware resources, one search user has no effect on another concurrent search user.

The rules for concurrent access to indexes are: users can share access to an index while searching, even if an index is being updated, merged, or compressed, but only one user at a time can have write access to an index. Therefore, when an index is being updated or compressed, no other users can modify that index. A "user" is a single process or thread. In a multi-threaded application, only one thread at a time can have write access to an index.   

A user who starts a search while an index is being updated will see the index as it existed before the update started. Once an index update commits, searches that begin after the commit will see changes made to the index by that update.  If an update is terminated abnormally, the index reverts to its state as of the last successful commit. Searches will remain unaffected by the failed update. At the start of the next index update, any invalid data added by the failed update will be removed before new data is added.

For information on resource contention issues when running multiple concurrent indexing operations, please see Optimizing indexing of large document collections.

Options

Option settings are maintained separately for each thread, so changes to options in one thread will not affect a job in progress on another thread. When a new thread is started, it will inherit the most recent option settings from other threads in the process.