How to integrate the dtSearch Engine with an external allocator such as the Intel TBB Scalable Allocator
In heavily multithreaded applications that do intensive memory allocation, heap contention can significantly affect performance, causing excessive amounts of time to be spent in the malloc() and free() functions. One way to address this issue is to use a replacement memory management library such as Intel's Scalable Allocator, tbbmalloc, or Microsoft's mimalloc. There is no single allocation strategy that is optimal for all possible usage patterns, so the best replacement memory management option will vary depending on the characteristics of each application.
Starting with version 7.97 of the dtSearch Engine, the 64-bit Windows version of the engine supports two options for using a replacement memory manager:
(1) Run-time override
Replacement memory managers usually have the ability to override the default allocation functions in a program and in the DLLs that it loads simply by being linked with the calling application, as long as the calling application and the related DLLs are dynamically linked with the C Runtime Library (CRT). Now that the 64-bit version of the dtSearch Engine for Windows (dtengine64.dll) uses the Microsoft Universal CRTs, in a C++ program the calling program can add a link dependency on the alternative memory management library such as tbbmalloc or mimalloc, causing it to be loaded at runtime. For example, the Intel tbbmalloc library can be linked with the program executable, causing malloc() and free() calls to be replaced with the corresponding tbbmalloc functions throughout the program, with no source code changes are needed.
(2) Linkage with the Intel TBB Scalable Allocator
An alternative build of the 64-bit dtSearch Engine that is itself linked with tbbmalloc as a replacement for malloc() and free() is also now included with the dtSearch Engine, in the C:\Program Files (x86)\dtSearch Developer\lib\engine\win\x64_tbb folder. This folder also includes tbbmalloc-linked versions of two external file parser DLLs, dtv_pdfcrypto64.dll and dtv_rar64.dll. Using this version of the dtSearch Engine provides a way to use the tbbmalloc library with the dtSearch Engine without the need to rely on the dynamic override mechanism, which can be problematic in some environments such as .NET Core.
In our testing, the Intel tbbmalloc library has worked well as a scalable allocator in applications that do heavily multithreaded indexing or searching. It is available as open source under the Apache license and also with commercial support from Intel. For more information on the Intel library, please see: