Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Perl 5.004_04 IPC: Passing data structures between processes

by Tanalis (Curate)
on Jul 13, 2005 at 09:49 UTC ( [id://474479]=perlquestion: print w/replies, xml ) Need Help??

Tanalis has asked for the wisdom of the Perl Monks concerning the following question:

I'm currently working on a script to access a remote database a number of times, downloading a number of sets of data and collating those datasets together.

The database is extremly large and extremely remote and hence access to it can be very slow (circa 60 seconds to download 1500 rows). To get around this, I'm looking at using a number of processes to download the datasets simultaneously, returning those datasets to the controlling process for collation and final processing.

I've written code to do this that creates the required number of processes and successfully downloads the datasets from the database as a reference to a hash of hashes.

What I'm having trouble doing (in any clean way) is returning those datasets to the controlling process. My current solution involves IO::Select, IO::Pipe and Data::Dumper, which works, but isn't scalable enough for my needs(1) - I'm testing using a subset of a much larger and much more complex dataset which I'll need to handle once I hit the production database. Using temporary files over pipes runs into the same issue.

Unfortunately, the shop I'm working for uses Perl 5.004_04 as standard, and we have some quite draconian restrictions regarding external code(2) - meaning that my preferred solution, using IPC::Sharable, isn't open to me. Using threads over processes would also get me around the problem, but again, isn't something that's available.

My train of thought at the minute is leading me to think that rolling my own XS module to allow me to access a C-style shared memory segment is probably the way to go.

Before I embark on that route, does anyone have any suggestions about a better way to achieve something like this?

Any thoughts would be appreciated :)

Replies are listed 'Best First'.
Re: Perl 5.004_04 IPC: Passing data structures between processes
by davorg (Chancellor) on Jul 13, 2005 at 10:42 UTC

    I'm sure that you've been told this before, but using such an old version of Perl with no CPAN modules is just madness. Why deliberately cripple yourself like that? You (and other programmers you work with) will reap major benefits if you spend some time getting that policy changed.

    Or, if that's really not an option then I seriously suggest you start looking for a new client or a new job (as appropriate)!

    --
    <http://www.dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

      I know that, you know that. This is an ongoing battle that's been going on for at least the past 2 years .. and shows no sign of getting sorted out yet.

      I can understand the Perl version, mind - it'll take a fairly hefty number of developer hours to go through all of the code we have that works around Perl 5.004 bugs (or, worse, fails to run on later versions of Perl) to fix issues.

      Unfortunately, in the "here and now", I'm stuck with what I have to work with. It's stupid, it's causing problems, but it's the way things are.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://474479]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2024-03-28 20:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found