I've solved a similar problem lately, namely importing a big-csv file of records into my system. I'll lign out my solution path, maybe it helps you.

The problem:

User should be able to upload a file that gets sanity checked and imported into a database. As such files can be very big, and the import can take long the user shall be presented a progress bar until the import has finished.

The idea:



The biggest part of the problem is to make the call to the importer "non blocking" so that the caller can end and send it's response. That's the reason why forking is not an option here. My first attempt to solve this was by using IPC::Open2, but that didn't work out.

I found the easiest way to do so is to use LWP::UserAgent with a timeout of 1. In my case, I've actually used an RPC call to a webservice, but the principle remains the same: no blocking by making an http-request and not waiting for the response.

So what we now have is a background-job on the server that does the importing. While running, this job will also update the progress counter file.

The last part of the setup is a little server component that will be passed a token and return the progress of that token (means it will read out the progress counter file and return the content as plain text).

This component will be called by the Javascript in order to update the progress bar.

The solution:

This is the import controller: 
sub csv : Local
{
    my ( $self, $c ) = @_;

    my $konf = ECS::Inkasso::Konfig->new($c);
    my $vid  = $c->req->{parameters}->{vorgang_id};
    
    my $to_in  = $konf->{location}->{csv_upload}."/incoming";
    my $to_ok  = $konf->{location}->{csv_upload}."/ok";
    my $to_err = $konf->{location}->{csv_upload}."/error";

    eval
    {
        if ( my $upload = $c->request->upload('my_file') )
        {
            
            $upload = FileSystemObjects::File->new( $upload->{tempname
+} );
            
            my $md5 = $upload->content_md5_file('hex');
            
            die "Diese Datei ist bereits in Bearbeitung.\n"
                if -e "$to_in/$md5";
                
            die "Diese Datei wurde bereits verarbeitet.\n"
                if -e "$to_ok/$md5";
                
            die "Diese Datei wurde bereits abgelehnt.\n"
                if -e "$to_err/$md5";
            
            my $xmlrpc = XML::RPC->new('http://localhost:8080');

            my $ping;
            
            eval { $ping = $xmlrpc->call( 'inkasso.ping', 3, {'wait'=>
+0 } ) };
            
            if ( $ping =~ /^\d+$/ )
            {
                $upload->move( "$to_in/$md5" );

                eval
                {
                    $xmlrpc->call( 'inkasso.csv_importieren', 1,
                    {
                        auth  => { id => 1, pwd => "derJochen" },
                        token => $upload->name,
                    });
                };

                $c->res->redirect($c->req->base."/forderung/importiere
+n/import_laeuft?token=".$upload->name);
            }
            else
            {
                die "Webservice nicht verfügbar.\n";
            }
        }
        else
        {
            $self->my_forward( $c, content_template => 'forderung/impo
+rtieren/csv' );
        }
    };

    if ( $@ )
    {
        $self->my_forward( $c, content_template => 'allgemein/error', 
+error_msg => $@ );
    }
}

sub import_laeuft : Local
{
    my ( $self, $c ) = @_;
    
    my $token = $c->request->{parameters}->{token} || die "no token!\n
+";
    
    die "malformed token!\n" 
        unless $token =~ /^[a-f0-9]+$/i;

    $c->stash->{token} = $token;

    $self->my_forward( $c, content_template => 'forderung/importieren/
+progress_bar' );
}

[download]
This is the html for the progress bar 
<h2>CSV - Import l&auml;uft</h2>
<p>Bitte warten Sie ab bis die Pr&uuml;fung abgschlossen ist und sie e
+in Ergebnis sehen.</p>
<p>Benutzen Sie dann die weiterf&uuml;hrenden Links.</p>
<table class="content">
    <tr>
        <td style="border:solid black 1px">
            <table style="border-collapse:true">
                <td id="progress_done" style="padding:0;margin:0;backg
+round-color:#006699;width:0px;height:10px;"></td>
                <td id="progress_left" style="padding:0;margin:0;backg
+round-color:#EEEEEE;width:400px"></td>
            </table>
        </td>
    </tr><tr>
        <td id="progress_label" style="text-align:center;"></td>
    </tr>
</table>
<script>
check_progress(
    '[% c.uri_for("/") %]', 
    '[% token %]', 
    'progress', 
    400
);
</script>

<div id="progress_ok" style="display:none">
<h2 style="color:green">CSV - Import erfolgreich.</h2>
<p>
Die Datei wurde erfolgreich gepr&uuml;ft und in das System eingestellt
+. <br />
Es kann jedoch einige Zeit dauern, bis die Daten verarbeitet sind.
</p>
<h3>Links</h3>
<a href="[% c.uri_for('/forderung/importieren/protokoll_anzeigen') %]?
+token=[% token %]&mode=ok">Protokoll anzeigen</a><br />
<a href="[% c.uri_for('/forderung/uebersicht/neue_forderungen') %]">&U
+uml;bersicht neue Forderungen</a><br />
</div>

<div id="progress_error" style="display:none">
<h2 style="color:red">CSV - Import fehlgeschlagen.</h2>
<p>
Die Datei enthielt mindestens einen fehlerhaften Satz und wurde zur&uu
+ml;ckgewiesen.<br />
Bitte &uuml;berpr&uuml;fen Sie das Protokoll und korrigieren sie Ihre 
+Daten.
</p>
<h3>Links</h3>
<a href="[% c.uri_for('/forderung/importieren/protokoll_anzeigen') %]?
+token=[% token %]&mode=error">Protokoll anzeigen</a><br />
<a href="[% c.uri_for('/forderung/importieren/csv') %]">nochmal probie
+ren</a><br />
</div>

[download]
This is the updater JS 
function check_progress (base_url, token, to, pwidth)
{
    poll_token(base_url, token, to, pwidth);
}

function poll_token (base_url, token, to, pwidth)
{
    var a = new Ajax.Request ( base_url + '/progress/get?token=' + tok
+en,
    {
        method: 'get', 
        onSuccess: function(t)
        {
            var s     = t.responseText.toString();
            
            if ( s.search(/^(\d+\.\d+)\t(.+)/) > -1 )
            {
                var a = s.match( /^(\d+\.\d+)\t(.+)/ );

                var prozent = a[1];
                var meldung = a[2];
                
                var p = 1 / (100/prozent);
                
                if ( pwidth == null )
                {
                    pwidth='200';
                }
                
                var l = pwidth * p;
                var r = pwidth * (1 - p);

                $(to + '_label').innerHTML = meldung + "( " + prozent 
++ "% )";
                $(to + '_done').style.width = l.numberFormat('#') + "p
+x";
                $(to + '_left').style.width = r.numberFormat('#') + "p
+x";

                var f = function () { poll_token(base_url,token,to, pw
+idth); };
                setTimeout(f,500);
            }
            else
            {
                if ( s.search(/^ok/i) > -1 )
                {

                    $(to + '_done').style.width = pwidth + "px";
                    $(to + '_done').style.background = 'green';
                    $(to + '_label').style.color = 'green';
                    $(to + '_label').innerHTML = "abgeschlossen ( 100%
+ )";

                    $(to + '_left').style.width = "0px";

                    $(to + '_ok').style.display = 'inline';
    
                    return true;
                }
                
                if ( s.search(/^error/i) > -1)
                {
                    $(to + '_done').style.background = 'red';
                    $(to + '_label').style.color = 'red';
                    
                    $(to + '_error').style.display = 'inline';
                    
                    return false;
                }
                
                var f = function () { poll_token(base_url,token,to, pw
+idth); };
                setTimeout(f,500);
            }
        }, 

        onFailure:function(t)
        {
            alert('Fehler beim Zugriff auf Webserver: ' + t.status + '
+ -- ' + t.statusText);
        }, 
        asynchronous:!check_sync
    });
}

[download]
And finally the server sided progress component 
package ECSInkassoWeb::Controller::Progress;

use strict;
use warnings;

use Data::Dumper;

use base qw(Catalyst::Controller ECSInkassoWeb::Controller::Base);

use FileSystemObjects::File;

sub get : Local
{
    my ( $self, $c ) = @_;
    
    my $token = $c->req->{parameters}->{token};

    my $konfig = ECS::Inkasso::Konfig->new($c);
    
    my $f = FileSystemObjects::File->new( $konfig->{location}->{progre
+ss} ."/$token" );
    my $h = $f->open("<");
    
    $_ = <$h>; chomp;
    
    $c->response->headers->header('content-type' => "text/plain; chars
+et=UTF-8");
    $c->response->body( $_ );
}

1;

[download]
The snippets are directly taken from the production server, and contain german. Let me now if that is problem.


holli, /regexed monk/

In reply to Re: Client Pull Not Working correctly by holli
in thread Client Pull Not Working correctly by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.