Multi-threading Net::LDAP

Cuhulain has asked for the wisdom of the Perl Monks concerning the following question:

Venerable Monks,

I have Perl 5.10 programs on Solaris 10 that use Net::LDAP to:

Bind to an X.500 Directory System Agent (DSA)

Update (Add or delete or modify) thousands of entries

Unbind

The mean time for the DSA to service each LDAP update is 51 miliseconds.
The mean time for my Perl program (LDAP client) to prepare the next LDAP update request is 13 milliseconds.

The resulting round time per update is 64 milliseconds, giving about 15 updates per second. I have tuned the DSA to about its optimal performance, according to the vendor. So I cannot reduce that 51 ms time per LDAP update by the server. However, I run the X.500 DSA as one parent process that spawns 10 children. So I am considering forking my LDAP client Perl program to split the possibly thousands of updates into up to 10 batches to be processed in parallel. Thus I hope to finish all updates in one tenth of the current time.

Can Monks kindly advise whether Net::LDAP provides any built-in support for multi-threading? Or should I somehow fork off 10 child processes and assume that each may bind to a different child LDAP server?

I'd welcome advice on either approach.

My first edition of Perl Cookbook has recipies in Chapter 17 Sockets on:

17.11 Forking Servers

17.12 Pre-Forking Servers

Should I adapt these Socket examples to LDAP, or should I look to more recent books/editions for improvement in threading support in Perl? I have subscribed to the Sourceforge LDAP mailing list, but its archives seem to be heavily spammed - I cannot get anything useful from the list archives.

Thank-you for any guidance you can offer.

Comment on Multi-threading Net::LDAP

Replies are listed 'Best First'.
Re: Multi-threading Net::LDAP by Eliya (Vicar) on Feb 15, 2012 at 10:12 UTC
If I'm understanding you correctly, you don't need a multi-process server (the cookbook section you mentioned), but a multi-process client, e.g. using Parallel::ForkManager. (Also, don't confuse threads with processes.)	[reply]
Re: Multi-threading Net::LDAP by locked_user sundialsvc4 (Abbot) on Feb 15, 2012 at 14:55 UTC
You could, with the appropriate use of two threads, overlap the 13ms with the 51ms, by virtue of preparing a handful of new requests in the same time that the request is being processed. Or, you could launch an asynchronous write, prepare the next request or two, and then stick your catcher’s mitt up into the air in time to catch the ball on its way down. Good ol’t `select()` can go a long way sometimes. But you still have to consider where is “the ruling constraint.” How fast can the target of your entreaties process requests? How many can it do “literally at the same time?” If it were by some means presented with 10 requests simultaneously (and from the same TCP/IP port address), could it actually process them simultaneously? It will do no good at all to generate a bunch of parallel requests “over here” only to find that they have to wait in line, e.g. waiting on some mutex or database lock or whatnot, “over there.” You need to find that out first, before you start writing very-clever software. (And especially before you actually say to anyone, such as your boss, “I think I know how I can make this thing run ten times faster,” because, once such perhaps-regretful words have passed your lips, you have ... Signed Up.™)