However, it seems from your description that the “unknown query” would parse/search on the value of each key (“search through a hash”) i.e. look at each value in turn for each query. I always think of a hash as a “lookup table” and it doesn't look like this is what you'll be doing. It maybe that an array would be more suitable, less overhead at least. Something like NAME1|SEQUENCE1. According to the docs D::D can build disc based arrays too although I have never tried.

How unknown are your unknowns? Could there be general subsets/groups that could be preprocessed; you say there is additional processing to be done on the results – could that be done upfront? A series of additional lookups that could identify a smaller group of records to search – divide and conquer? If you did choose the RDBMS route are there indices that could be built that would help find what you're after. If there aren't it would, imo, likely negate any advantage.

If you have to process each record for each search then whatever method you adopt is going to be slow and possibly not suitable for a website.

If you could up come with an outline of any known unknowns there is a good chance the monks could give you some pointers on how to avoid what seems like a brute force approach. It would be a challenge many of them would be unable to resist. :-)

I think you are reading too much into his use of the word "search", but you have a point there. If he ever wants to do lots of searches for parts of a key (i.e. "contains" instead of "equals"), a hash will be as bad as a simple array.

Wait, is the program called BLAST something you already have or is that the script you want to create? I.e. why would you use BLAST to search and then use a different script for retrieval? Obviously when the retrieval script creates that output file the number of hits is available with zero effort

If you want to use your script with a webserver you probably will observe the following with the different solutions:

A) perl CGI script, loading hash in memory: Really bad, every search has to load the database into memory. Even if that takes only 4 seconds it is mayor disk wear and slow. Same with Storable

B) Using mod_perl, loading hash into memory: With a trick (using a BEGIN block to load the hash) you could read the data only once into memory. Big improvement on execution speed, but memory could be a problem as apache usually creates multiple instances of the web server each running the script and data. Depends on the server engine you use, a threaded engine probably could share the data but not a forking engine (Me no expert here, someone hopefully will correct me if I'm wrong). Same with Storable

C) Using SQLite, DBM::Deep, ...: Disk based methods. Data is not loaded into memory, an index keeps retrieval reasonably fast. Obviously somewhat slower than a simple hash in memory, but still simple to program. No problem if your database grows in size (within limits), no noticable startup time. If the database does mostly reads and seldom writes arguably the best solution in your case IMHO

D) Using mysql or postgresql: Apart from the web server you have to install and manage another server (easy on linux (easy for me, not at all easy for my mother ;-) ), not so easy on windows(?)). Reading and changing data in the database never is a performance problem irrespective of size. Complex databases possible (which you don't seem to need), the database server will get bored to death in your case ;-). Probably a notch slower than solution C, but the only solution if you want 100 simultaneous apache threads reading and writing to that database.