I am developing an app that is very DB intensive. By very I mean totally. I am currently caching all the static data I can. Although the DB queries are non trivial the return value I want is - it's always a simple integer. An ideal case for a hash table cache. A complicating factor is that there are about 50 parallel processes all doing the same thing to handle the load so the cache needs to be shared between processes, hence the desire to use shared memory to do it.
Not being one to reinvent the wheel I have looked at everything available on CPAN - basically Cache::Memory Cache::Memory::SharedMemory and the underlying IPC::ShareLite module.
The main problem with what is available is that the shared Memory module Cache::Memory::SharedMemory is glacial. While it works benchmarking shows it to be at 2-3 orders of magnitude slower than Cache::Memory. In fact it is way slower than raw DB access which of course defeats the purpose. If anyone knows of an existing FAST way to cache SQL query results and share that cache I am all ears.
Because of the issues with speed, unwanted data serialization, inefficient data expiry and no easy way to limit memory useage I have decided I probably need to roll my own module that does what I want - namely use a closed hash table to store hashed SQL along with the corresponding integer result. Naturally as speed is the issue this will essentially be in C with just enough Perl for the interface. To take advantage of CPU caching and place a finite limit on memory useage I plan to store the hash key, insertion time and data sequentially in memory. I also don't plan on having explicit linked list collision handling, simply letting new coliding data overwrite the old will be sufficient (the lost key value then has to come from the DB which is fine - I am storing the full hash for validation purposes). The trivial data size allows a wide (essentailly inefficient hash table) where memory is traded off for speed as always.
Anyway enough background. First I am unfamiliar with shared memory (but learning fast). What I would like to know is if I really have to lock access to memory locations while they are written given a data stuct width of (probably 12 bytes) (long ints for hash key, time, and data) and the potential consequences of not doing so. The app runs on Linux (so single threaded multi-process but on a multiprocessor machine) so it is theoretically possible that two processes could try to write/write write/read read/read to the same location ?literally simultaneously - at least as I understand it. Data corruption is tolerable to a degree as corrupting the hash key would effectively return a cache miss which is fine. Crashing the system is another issue. Any advice most welcome.
cheers
tachyon
s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print
In reply to System V shared Memory Perl Module (more of a C question than Perl) aka Efficient SQL result caching by tachyon
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |