Your skill will accomplish what the force of many cannot |
|
PerlMonks |
Perl 5.004_04 IPC: Passing data structures between processesby Tanalis (Curate) |
on Jul 13, 2005 at 09:49 UTC ( [id://474479]=perlquestion: print w/replies, xml ) | Need Help?? |
Tanalis has asked for the wisdom of the Perl Monks concerning the following question:
I'm currently working on a script to access a remote database a number of times, downloading a number of sets of data and collating those datasets together.
The database is extremly large and extremely remote and hence access to it can be very slow (circa 60 seconds to download 1500 rows). To get around this, I'm looking at using a number of processes to download the datasets simultaneously, returning those datasets to the controlling process for collation and final processing. I've written code to do this that creates the required number of processes and successfully downloads the datasets from the database as a reference to a hash of hashes. What I'm having trouble doing (in any clean way) is returning those datasets to the controlling process. My current solution involves IO::Select, IO::Pipe and Data::Dumper, which works, but isn't scalable enough for my needs(1) - I'm testing using a subset of a much larger and much more complex dataset which I'll need to handle once I hit the production database. Using temporary files over pipes runs into the same issue. Unfortunately, the shop I'm working for uses Perl 5.004_04 as standard, and we have some quite draconian restrictions regarding external code(2) - meaning that my preferred solution, using IPC::Sharable, isn't open to me. Using threads over processes would also get me around the problem, but again, isn't something that's available. My train of thought at the minute is leading me to think that rolling my own XS module to allow me to access a C-style shared memory segment is probably the way to go. Before I embark on that route, does anyone have any suggestions about a better way to achieve something like this? Any thoughts would be appreciated :) -- Foxcub
Back to
Seekers of Perl Wisdom
|
|