vonersnae has asked for the wisdom of the Perl Monks concerning the following question:

Hello all, I would like to ask for some help, if you don't mind :)

Anyway for my problem, I'm currently building a script that should be capable of doing a simultaneous request to a certain server. Just like 10 individuals accessing a specific site at the same time.

Im using thread, Thread::Pool module for that matter, WWW::Mechanize to do the requesting stuff. I don't like to destroy the session of every request, that's why I created a loop that runs all throughout until interrupted. So basically every thread keeps on throttling until interrupted.

Example code:

#!/usr/bin/perl -w use strict; use Thread::Pool; use WWW:Mechanize; my %check :shared = ('success', 0, 'fail', 0, 'exit', 0); my $pool = Thread::Pool->new( { optimize => 'memory', workers => 1, do => \&request, } ); for(1..10){ $pool->job(); $pool->add; { print "Please type 'exit' to quit: "; chomp ($_=<STDIN>); if($_=~/exit/i){ $check{exit} = 1; $pool->shutdown;} print "success: $check{success}\tfailed: $check{fail}\n"; sub request{ my $flag = 0; my $mech = WWW::Mechanize->new(autocheck => 1); $mech->get('http://example_server.com/login.php'); $mech->submit_form( form_number =>1, fields =>{ username => 'testuser' password => '123qwe' } ); if($mech->uri() =~/home/i){ #check if login if successful $check{success}++; $flag = 1; } else { $check{fail}++;} while($check{exit} != 1 && $flag == 1){ $mech->follow_link( url_regex => qr/nextpage/i); $mech->get('http://example_server.com/home'); } }


Question:

- Is threading the appropriate tool to use or should I use fork instead?
- Just an example, if the server's maximum number of simultaneous requests is set to 10, If the script is not interrupted I should not be able to access the server using my browser right? if the code is correct.
- Memory wise, is there any way I can alleviate the memory usage?

-= thanks for your help...

Replies are listed 'Best First'.
Re: Simultaneous request using threading
by stiller (Friar) on Apr 11, 2008 at 07:50 UTC
    - Is threading the appropriate tool to use or should I use fork instead?
    I tend to prefer fork, but threads might be ok

    if the server's maximum number of simultaneous requests is set to 10, If the script is not interrupted I should not be able to access the server using my browser right?
    No. The server will serve up to ten requests at any time. But it won't take long to serve login.php, and once it's done with that, a slot is freed, for the next request to come in. You with a browser, or your 10-thread script might be the next to be served.

    - Memory wise, is there any way I can alleviate the memory usage?
    If you think your threaded script use much memory, try with forking. I think that forking usually consumes less memory, but I don't know if that is universally true, or if it's improved in 5.10
Re: Simultaneous request using threading
by BrowserUk (Patriarch) on Apr 11, 2008 at 08:38 UTC
Re: Simultaneous request using threading
by Sinistral (Monsignor) on Apr 11, 2008 at 14:41 UTC

    Based on this description of your needs:

    Anyway for my problem, I'm currently building a script that should be capable of doing a simultaneous request to a certain server. Just like 10 individuals accessing a specific site at the same time.

    I think you should run, do not walk, and check out LWP::Parallel

Re: Simultaneous request using threading
by vonersnae (Novice) on Apr 12, 2008 at 03:03 UTC
    ** No. The server will serve up to ten requests at any time. But it won't take long to serve login.php, and once it's done with that, a slot is freed, for the next request to come in. You with a browser, or your 10-thread script might be the next to be served.

    - wooot!, thanks a lot thats very helpful, just a quick question though. Is there a way in Apache(the server) to limit the number of clients logged-in? I mean, like some sort of a setting that sets the maximum number of active client/active session(not the request itself).

    ** Threads could work for you, but the first thing to do is drop Thread::Pool like a hot brick. It is just complexity and overhead with no benefit.

    - OK, ill try that, I hope it will help me alleviate the memory usage

    ** I think you should run, do not walk, and check out LWP::Parallel

    - nice nice, interesting module, damn how come i missed this module hahaha, thanks!

    Just a clarification:

    - Basing on my code above, I don't know if the session is still active every time it spawns another thread. I just created a loop of requests hoping it will be active until interrupted, its only a theory, no proof. So im asking a clarification from you guys if what do you think, is it still active? or my code just simply sucks haha.

    Now the question, how can I prove that the session is still active? logs? packet sent?

    Just incase you're wondering why on earth am I making this script...Its because I was given a task to test the server's capability on handling every client. To check if how the server is responding if there are a number of clients accessing it simultaneously. Example: 20 clients accessing the server, how will the server react, does it still runs on its full potential...

    By the way, cookie used, not persistent one. So every time the browser is closed...cookie expired..
Re: Simultaneous request using threading
by vonersnae (Novice) on Apr 13, 2008 at 05:29 UTC
    Opps..that Anonymous just posted a reply is me hehe. I forgot to login haha....well again, 1Gig of memory usage on 100 requests, expected or not?
Re: Simultaneous request using threading
by Anonymous Monk on Apr 13, 2008 at 05:18 UTC
    I did some research and found some interesting module like what Mr. Sinistral gave me the LWP::Parallel. I also managed to stumble on some of the threads here in Perlmonks that discussed about my concern. I'll just continue reading this stuff and try to figure out how am I going to lessen the memory usage, because when ever I tried to spawn 100 requests, it surely goes way way up high, 1Gig of memory usage.

    I'm also thinking of using httperf, well anyway, thanks for your help guys...

    and by the way, 1Gig of memory usage on 100 requests, expected or not?