The following are some option settings that can be used to reduce the size of a database index.
Binary Files
The binary files option setting controls the way dtSearch treats files in a format that it does not recognize as a document. There are three options: (1) index the files completely, (2) filter out only the text of the files, and (3) skip the files entirely.
Filtering or skipping binary files can greatly reduce index size, and improve indexing speed. Indexing binary files completely can have a large effect on index size because binary files do not contain a normal mix of words. If treated as text, they produce a large number of unique, random text sequences, which then bloat the word list portion of the index disproportionately.
Filtering is also more effective at extracting text than simply indexing binary files entirely. This is because text may be present in a variety of formats (for example, blocks of Unicode text are often mixed with blocks of single-byte text), and the filtering algorithm can identify and decode these segments.
Exclude filters are another good way to minimize the number of binary files included in an index.
Title Size
If the documents being indexed are small, the per-document overhead in an index may be a relatively large portion of the total index size. For each document, dtSearch stores the filename and location, the modification date, size, and other properties, as well as a "title" which is usually the first 80 characters of text from the file. To reduce the size of the index, the title size can be changed to a smaller value. Additionally, reductions in the size of the filenames in the index, including the folder name, will save space.
Noise Words
For text files, a noise word list can reduce the size of an index by eliminating common words like "the" or "if". By default, dtSearch will index documents using a noise word list for the English language. The noise word list is a plain text file named noise.dat that can be edited using Notepad to add additional words.
Numbers
dtSearch has an indexing option to skip indexing numbers. If the documents being indexed contain many numbers, and if these numbers do not have to be searchable, this setting can reduce index size considerably.
Hyphens
The option setting to treat hyphens as spaces produces smaller indexes than any of the other option settings.
Reagrds
Anup.
Bluesky
