Re^2: Large Constant Database in Text File

Thank you - 1 second searches sounds rather good!

I have to say that its noticeable that the 2 responses that talk performance come from people comfortable with a non-relational approach...

I'd be interested to hear from any of the relational database proposers of an equivalent relational database based solution to what's here and the performance delivered.

I'll have to study the code because I'm not as good at Perl as I'd like to be but I much appreciate the working sketch and it looks like a solution is possibly easier to achieve than I imagined. Thanks once again.

Comment on Re^2: Large Constant Database in Text File

Replies are listed 'Best First'.
Re^3: Large Constant Database in Text File by BrowserUk (Patriarch) on Sep 04, 2006 at 16:02 UTC
The bottom line is that the data and indexes reside on a disk somewhere. Your process can either: Load them itself, follow the pointers and extract the required data. Or: Compete for attention from a DB process. Wait for connection. Negotiate authorisation. Wait while the SQL parser parses your query. Wait while the DBM checks your query against the schema. Wait while the DBM finds the relevant data and indexes. Wait while the DBM optimises your query. Wait while it checks/frees cache space for your indexes. Wait while it actually runs the query and builds a results table. Wait while the communications negotiate the transfer of the results. Wait while your process allocates space to hold the results. Wait while it structures the results into a perlish data structure. Finally, do something with the results if you haven't run out of memory in the meantime. There are many good reasons for using an RDBMS, not least of which is that someone else does a lot of the work and maintenance, but performance is not one of them! Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]

Replies are listed 'Best First'.

Re^3: Large Constant Database in Text File
by BrowserUk (Patriarch) on Sep 04, 2006 at 16:02 UTC

The bottom line is that the data and indexes reside on a disk somewhere. Your process can either:

Load them itself, follow the pointers and extract the required data.
Or:
1. Compete for attention from a DB process.
2. Wait for connection.
3. Negotiate authorisation.
4. Wait while the SQL parser parses your query.
5. Wait while the DBM checks your query against the schema.
6. Wait while the DBM finds the relevant data and indexes.
7. Wait while the DBM optimises your query.
8. Wait while it checks/frees cache space for your indexes.
9. Wait while it actually runs the query and builds a results table.
10. Wait while the communications negotiate the transfer of the results.
11. Wait while your process allocates space to hold the results.
12. Wait while it structures the results into a perlish data structure.
13. Finally, do something with the results if you haven't run out of memory in the meantime.

There are many good reasons for using an RDBMS, not least of which is that someone else does a lot of the work and maintenance, but performance is not one of them!

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]