Hi,

My guess is that the files are multimedia of some sort. If this is a search of the binary data that will occur on a reoccurring basis, you might as well go with MySQL or some other DBMS.

As this appears to be regressive search pattern, you will have to determine how many iterations of the 4 bytes you are going to store in the database. Preferrably you will want enough iterations stored for the file record to be as unique as possible. There really isn't an answer that any of us here can give you as to how many iterations... as it will depend highly on the files and the search patterns you are generating with the application.

In DBMS terms, you are looking to create indexes of binary files. NONE of the DBMSs do this directly themselves (text files and limited other documents .. yes) so you will have to spend some time and design the 'homemade' index.. naturally, you will have an index on this index to avoid table scanning. My recommendation is to store the 4 byte chunks (nibble) as characters rather than actual byte(4) as you will be able to create useful indexes on the columns.

You will have to make a trade off... whether you want it to be painful (slow) to add another binary file (scan the existing table entries - not files - for duplicate nibbles.. if duplicate entry exists, add another nibble to the search pattern and retest the new against the 'duplicate'.),
or painful on the main search (all entries will have say '10' nibbles stored in the dbms - scanning two entries that are duplicates would require reading of the two files in question - more i/o on the read)

generally if your users are going to be primarily reading the documents and not adding new ones, then make the insert more i/o intensive - you will save time on the retrieval.

generally if your users are going to be primary using it for storing, say video files, then a hard coded nibble count is probably best.

you will, of course, test both methods (and probably others) to determine which is best... or you could make a hybrid system but design, testing and maintenance is likely going to be much higher as the complexity increases.

You may want to consider using one of the AI perl modules or similar to attempt to find recognizable patterns that your application generates...

Storing the files (blobs) in the database is generally a bad idea as it neither MySQL or any of the other DBMSs have ever been very good at processing and storing large BLOBS with any reasonable amount of performance hit in comparison of storing them on the filesystem itself. Small blobs are okay, but remember that the larger the data row (or text/image datatype if you are used to Sybase) the more physical and logical i/o is required. You will always want to avoid unnecessary i/o (logical or physical).

If these are multimedia files, you might be able to get away with simply storing the metadata (name, video codec used, audio codec used, author, etc.)... it probably would be a good idea even if you don't plan on searching on the meta data just yet... that way you won't have to recreate the table definitions, etc.

Jason L. Froebe

Whether you end up using Perl or C should hinge on how comfortable you are with using "pack/unpack" vs. using ints or structs in C.

jfroebe seemed to be familiar with what you might be doing, but I was puzzled by your description:

I need to be able to search through one of the files in such a way that it takes 4 bytes, checks for a match, take the next 4 bytes if it's found it, or if not skip 4 bytes and repeat.

Well, whatever the nature of the search task, I think the better approach to indexing and searching the file data would be something like this:

• partition the 100K files into a sensible directory tree, to strike a good balance between tree depth and number of entries per directory (e.g. 40 directories, each with 50 subdirectories, each with 50 files, or maybe even something that represents "semantic" differences in file content, such as date, source or whatever).
• use a database (SQL-based RDBMS or even any sort of DBM file approach) to store tuples of "index_data, path/filename", where the index data (the search/key value) is the minimum needed to uniquely identify the content of each file.

Of course, if any of the files differ only slightly in their contents -- and only near the end -- you may need to divide the search into "stages". You would group files into directories based on similarity, so that the first one, two or three stages of the search/match process serve to select the proper directory path, and the final stage of searching is only worrying over a small number of files.

I can't make any more detailed suggestions because I don't understand the task well enough. In fact, I'm wondering if my suggestions are completely off the mark.