Used by dtsIndexJob to index non-file data that is passed to the indexer through callbacks.
File: dtsviewr.h
|
Data Member |
Description |
|---|---|
|
Get the document whose properties were just returned by getNextDocInfo or getDocInfoByName | |
|
Pointer to a function that will return information about a named document in the data source. | |
|
Pointer to a function that will return the next document in the data source as a dtsInputStream. | |
|
Pointer to a function that will return information about a the next document in the data source. | |
|
Pointer to user data (generally used to hold the "this" pointer for your object that implements the dtsDataSource). The rewind() and getNextDoc functions will receive the pData pointer as the first parameter on each call. | |
|
Provides information on the most recently-indexed file | |
|
Pointer to a function that will initialize the dtsDataSource so the next getNextDoc call will return the first document. The rewind function should return 0 if the data source was succesfully initialized, or -1 if initialization failed. If rewind returns -1, nothing will be indexed from the data source. |
|
Data Member |
Description |
|---|---|
|
Get the document whose properties were just returned by getNextDocInfo or getDocInfoByName | |
|
Pointer to a function that will return information about a named document in the data source. | |
|
Pointer to a function that will return the next document in the data source as a dtsInputStream. | |
|
Pointer to a function that will return information about a the next document in the data source. | |
|
Pointer to user data (generally used to hold the "this" pointer for your object that implements the dtsDataSource). The rewind() and getNextDoc functions will receive the pData pointer as the first parameter on each call. | |
|
Provides information on the most recently-indexed file | |
|
Pointer to a function that will initialize the dtsDataSource so the next getNextDoc call will return the first document. The rewind function should return 0 if the data source was succesfully initialized, or -1 if initialization failed. If rewind returns -1, nothing will be indexed from the data source. |
The dtsIndexJob structure provides two ways specify the data to index: by files (the toAdd member) and by data source (the dataSourceToIndex member). Most commonly, the text exists in disk files, in which case you would specify the files to be indexed using the toAdd member, which provides ways to specify the directories and files you want to index. In some situations, however, the text to be indexed may not be readily available as disk files. For example, the text may exist as records in a remote SQL database. You could copy the text from the database to local disk files and index the local disk files, but the dtsDataSource API provides a more direct and efficient solution. To supply this text to the dtSearch indexing engine, you create an object that accesses the text and then attach a dtsDataSource describing the object as the dataSourceToIndex member of a dtsIndexJob.
A dtsDataSource is a structure that provides access to any source of text data divided into logical documents. It consists of a set of function pointers for functions to retrieve documents and to iterate over the data to be indexed. The simplest possible data source would implement two function pointers: rewind, to initialize the data source, and getNextDoc, to get the next document to index. A dtsDataSource also contains a pData pointer that will be passed to the rewind() and getNextDoc() functions. To create a data source, you would do the following:
- Create an object that access the data source.
- Set the pData member of a dtsDataSource to point to this object.
- Make callback functions that convert the pData pointer to the object type and then call the appropriate member function, and pass those callback functions to the dtSearch engine in the dtsDataSource.
Your data source object (the CMyDataSource object in the example below) will return logical documents using another structure, a dtsInputStream. For each document, dtsInputStream provides its filename (any legal Windows filename), creation and modification dates, and size as it will be recorded in the index. Like dtsDataSource, dtsInputStream relies on callback functions -- seek() and read() -- to provide access to the data to be indexed.
dtSearch indexes the dtsInputStream objects created by the dtsDataSource by calling their seek() and read() members to get text data. After a dtsInputStream has been indexed, dtSearch will destroy it through a call to its release() member. dtSearch does not delete the dtsDataSource object so the caller is responsible for disposing of it when release() is called.
To avoid requiring an unnecessary initial pass through the input data before indexing, dtsDataSource knows nothing about the total size of the data to be indexed or the number of documents to be indexed. As a result, the dtSearch Engine will not be able to report the percentage completion of an indexing job involving a dtsDataSource.
For an example demonstrating use of the dtsDataSource API, see dsource.cpp.
Once a data source has been indexed, subsequent updates will be faster if the data source can quickly identify which documents have been modified since the last update, and which documents no longer exist. To make incremental updates more efficient, a data source can implement three additional functions: getNextDocInfo, getCurrentDoc, and getDocInfoByName.
getNextDocInfo and getCurrentDoc allow dtSearch to separate getNextDoc into two calls, one to get the document properties, and a second to get the document itself. The advantage of doing this is that dtSearch can skip the second step for documents that have already been indexed. getNextDocInfo returns information about the next document, most importantly the name and modification date. dtSearch uses this to determine if the document needs to be indexed, by comparing the name and modification date with what is already in the index. If the document is not in the index, or an older version is in the index, dtSearch will request the document contents in a call to getCurrentDoc. If the document does not have to be indexed, dtSearch will skip the document and call getNextDocInfo again for the next document's name and modification date. When a getNextDocInfo pointer is provided, getNextDoc will never be called, and dtSearch will rely only on getNextDocInfo and getCurrentDoc.
getDocInfoByName lets dtSearch check whether a particular document still exists, for purposes of implementing the "Remove Deleted" step in an index update.