If possible, run a quick stress test for your own peace of mind.
For example, you could run a test where you tie many DB_Files (i.e. much greater than 8) and randomly add/remove many records (i.e. much greater than 15000) to each one. While this is running, watch the resources used (CPU and disk utilization, etc.).
If you know DB_File performs ok under worse conditions than you expect, it will probably be a good indicator on how it will perform under expected conditions. If it doesn't perform well, you can cut back the numbers until you find out what its limits are and then plan to redesign your program accordingly.
| [reply] |
I don't think there are any limits from the Perl side.
However, Perl is using the db libraries present on your
system, and depends on their mercy. If there's a limit
in the library you use, your program might hit the limit.
Same for the overhead. There's some tie based overhead on
the Perl site, and some overhead because it needs to defer
actions to the library, but the juice is happening in the
library.
Abigail | [reply] |
Also keep in mind that both Perl and your db libraries are
going to be constrained by the system's per-process file-descriptor limit (usually not a problem unless you're talking hundreds or thousands of open files; see your shell's documentation on ulimit for how to check or alter it).
| [reply]
[d/l] |
I'm currently writing a module for creating read and write locks for these types of databases. Although I'm currently writting it for MLDBM d/bases using DB_File, I'll probably make it more adaptable for my own future use as well.
It just allows you to create a read/write lock, read or write your data and then flush it to the database without anyone contending your access.
I've got it working and have test run it concurrently in an ending loop with 3 other processes to see if the read and write locks hold and release and to make sure it deals with contention properly. I just need to tidy it up a bit and then I'll post it here for you to use if you like.
Cheers,
Dean | [reply] |
May I ask *why* swapping to SQL is not an option? (Even if you have a good reason, I'd like to know -- sometimes these types of questions start excellent threads.)
If it's a full-blown SQL database that you'd like to avoid, you might want to take a gander at SQLite. It's an SLQ database, but it's minimalistic, easy to install, and all data lives in one file on the filesystem. Pretty cool for those 'too-big-but-not-big-enough' problems. | [reply] |