Hi,

Background:
I am working on a replication project, where i have to replicate dependencies of projects. I use rsync command to replicate dependencies. Since the dependency list of a project varies and will usually have a huge list, hence i want to it replicate parallely by giving a bunch of file-names to replicate to each rsync command(using --files-from=<filename>) through a file. I am doing the same using perl's threads module.

Problem:
When i have more than one project(a common case), i create 10 threads to run rsync commands. This way i want to achieve replication of all dependencies parallely. I wait in main thread to finish all child threads using join() function of threads module. The script was running fine for somedays, but recently the script terminates, saying 'Segmentation fault', no line number is printed.

I have tried to debug(using perl -d <script-name>) the script, but the debugger does not comeup or no response when it lands on statement to create thread(threads->create(...)), i had to wait long time but i did not comeup so i have to kill the process.

I cannot post the whole code, because it is big and uses some of private packages, but here is how i create 10 threads for each project.

The script terminates by printing Segmentation fault, after executing last print statement in _replicate function. I not able to make out where and why that error is occuring. Or is there a other way to achieve parallel processing withoug using threads? Could someone help me here please?

I am using perl, v5.8.3 built for x86_64-linux-thread-multi 

sub _replicate{
    my $ref = shift;
    my $logger = get_logger();
    print "Starting replication of dependency files", $/;

    foreach my $sc(@{$ref}){
        next unless (defined $sc);
        
        mkdir($LOG_FOLDER."/".$sc->{sc_name});
        print "\tScenario: ".$sc->{sc_name}, $/;
        print "\tLatest Dependencies: ".$sc->{total_dep}." of size "._
+get_readable_size($sc->{total_size}), $/;
        my @thr_arr = ();
        print "Creating parallel threads", $/;
        
        foreach my $robj(@{$sc->{rsync}}){
            my $th = threads->create(\&worker, $robj); # i create thre
+ads this way
            push @thr_arr, $th;
        }
        #$logger->info("\twaiting for threads to finish its job...");
        print "\twaiting for threads to finish its job...", $/;
        foreach my $t(@thr_arr){
            if (defined $t){
                my $k = $t->join();  # this is how i wait for all thre
+ads to finish 
            }
        }
        #map {my $k = $_->join} threads->list;
        # map{
            # my $th = $_;
            # my $k = $th->join if($th);        # just a blind belief 
+whether this might cause 'Segmentation fault', hence the check.
        # }@thr_arr;
        #$logger->info("\tFinished replicating dependencies of ".$sc->
+{sc_name});
        print "\tFinished replicating dependencies of ".$sc->{sc_name}
+, $/;
    }
}

sub worker{
    my $robj = shift;
    my ($rsync, $server, $from, $to) = @{$robj->{elements}};
    my $alt_server = $RSYNC_CONN_STR_2;

    my $rsync_cmd = $rsync.$server.$from.$to;
    print "Thread-",threads->self->tid," executing ", $rsync_cmd;
}

[download]

Thanks in advance,
katharnakh.

In reply to Segmentation fault: problem with perl threads by katharnakh

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.