in reply to Client Pull Not Working correctly

I've solved a similar problem lately, namely importing a big-csv file of records into my system. I'll lign out my solution path, maybe it helps you.

The problem:

User should be able to upload a file that gets sanity checked and imported into a database. As such files can be very big, and the import can take long the user shall be presented a progress bar until the import has finished.

The idea:



The biggest part of the problem is to make the call to the importer "non blocking" so that the caller can end and send it's response. That's the reason why forking is not an option here. My first attempt to solve this was by using IPC::Open2, but that didn't work out.

I found the easiest way to do so is to use LWP::UserAgent with a timeout of 1. In my case, I've actually used an RPC call to a webservice, but the principle remains the same: no blocking by making an http-request and not waiting for the response.

So what we now have is a background-job on the server that does the importing. While running, this job will also update the progress counter file.

The last part of the setup is a little server component that will be passed a token and return the progress of that token (means it will read out the progress counter file and return the content as plain text).

This component will be called by the Javascript in order to update the progress bar.

The solution:

This is the import controller: 
sub csv : Local
{
    my ( $self, $c ) = @_;

    my $konf = ECS::Inkasso::Konfig->new($c);
    my $vid  = $c->req->{parameters}->{vorgang_id};
    
    my $to_in  = $konf->{location}->{csv_upload}."/incoming";
    my $to_ok  = $konf->{location}->{csv_upload}."/ok";
    my $to_err = $konf->{location}->{csv_upload}."/error";

    eval
    {
        if ( my $upload = $c->request->upload('my_file') )
        {
            
            $upload = FileSystemObjects::File->new( $upload->{tempname
+} );
            
            my $md5 = $upload->content_md5_file('hex');
            
            die "Diese Datei ist bereits in Bearbeitung.\n"
                if -e "$to_in/$md5";
                
            die "Diese Datei wurde bereits verarbeitet.\n"
                if -e "$to_ok/$md5";
                
            die "Diese Datei wurde bereits abgelehnt.\n"
                if -e "$to_err/$md5";
            
            my $xmlrpc = XML::RPC->new('http://localhost:8080');

            my $ping;
            
            eval { $ping = $xmlrpc->call( 'inkasso.ping', 3, {'wait'=>
+0 } ) };
            
            if ( $ping =~ /^\d+$/ )
            {
                $upload->move( "$to_in/$md5" );

                eval
                {
                    $xmlrpc->call( 'inkasso.csv_importieren', 1,
                    {
                        auth  => { id => 1, pwd => "derJochen" },
                        token => $upload->name,
                    });
                };

                $c->res->redirect($c->req->base."/forderung/importiere
+n/import_laeuft?token=".$upload->name);
            }
            else
            {
                die "Webservice nicht verfügbar.\n";
            }
        }
        else
        {
            $self->my_forward( $c, content_template => 'forderung/impo
+rtieren/csv' );
        }
    };

    if ( $@ )
    {
        $self->my_forward( $c, content_template => 'allgemein/error', 
+error_msg => $@ );
    }
}

sub import_laeuft : Local
{
    my ( $self, $c ) = @_;
    
    my $token = $c->request->{parameters}->{token} || die "no token!\n
+";
    
    die "malformed token!\n" 
        unless $token =~ /^[a-f0-9]+$/i;

    $c->stash->{token} = $token;

    $self->my_forward( $c, content_template => 'forderung/importieren/
+progress_bar' );
}

[download]
This is the html for the progress bar 
<h2>CSV - Import l&auml;uft</h2>
<p>Bitte warten Sie ab bis die Pr&uuml;fung abgschlossen ist und sie e
+in Ergebnis sehen.</p>
<p>Benutzen Sie dann die weiterf&uuml;hrenden Links.</p>
<table class="content">
    <tr>
        <td style="border:solid black 1px">
            <table style="border-collapse:true">
                <td id="progress_done" style="padding:0;margin:0;backg
+round-color:#006699;width:0px;height:10px;"></td>
                <td id="progress_left" style="padding:0;margin:0;backg
+round-color:#EEEEEE;width:400px"></td>
            </table>
        </td>
    </tr><tr>
        <td id="progress_label" style="text-align:center;"></td>
    </tr>
</table>
<script>
check_progress(
    '[% c.uri_for("/") %]', 
    '[% token %]', 
    'progress', 
    400
);
</script>

<div id="progress_ok" style="display:none">
<h2 style="color:green">CSV - Import erfolgreich.</h2>
<p>
Die Datei wurde erfolgreich gepr&uuml;ft und in das System eingestellt
+. <br />
Es kann jedoch einige Zeit dauern, bis die Daten verarbeitet sind.
</p>
<h3>Links</h3>
<a href="[% c.uri_for('/forderung/importieren/protokoll_anzeigen') %]?
+token=[% token %]&mode=ok">Protokoll anzeigen</a><br />
<a href="[% c.uri_for('/forderung/uebersicht/neue_forderungen') %]">&U
+uml;bersicht neue Forderungen</a><br />
</div>

<div id="progress_error" style="display:none">
<h2 style="color:red">CSV - Import fehlgeschlagen.</h2>
<p>
Die Datei enthielt mindestens einen fehlerhaften Satz und wurde zur&uu
+ml;ckgewiesen.<br />
Bitte &uuml;berpr&uuml;fen Sie das Protokoll und korrigieren sie Ihre 
+Daten.
</p>
<h3>Links</h3>
<a href="[% c.uri_for('/forderung/importieren/protokoll_anzeigen') %]?
+token=[% token %]&mode=error">Protokoll anzeigen</a><br />
<a href="[% c.uri_for('/forderung/importieren/csv') %]">nochmal probie
+ren</a><br />
</div>

[download]
This is the updater JS 
function check_progress (base_url, token, to, pwidth)
{
    poll_token(base_url, token, to, pwidth);
}

function poll_token (base_url, token, to, pwidth)
{
    var a = new Ajax.Request ( base_url + '/progress/get?token=' + tok
+en,
    {
        method: 'get', 
        onSuccess: function(t)
        {
            var s     = t.responseText.toString();
            
            if ( s.search(/^(\d+\.\d+)\t(.+)/) > -1 )
            {
                var a = s.match( /^(\d+\.\d+)\t(.+)/ );

                var prozent = a[1];
                var meldung = a[2];
                
                var p = 1 / (100/prozent);
                
                if ( pwidth == null )
                {
                    pwidth='200';
                }
                
                var l = pwidth * p;
                var r = pwidth * (1 - p);

                $(to + '_label').innerHTML = meldung + "( " + prozent 
++ "% )";
                $(to + '_done').style.width = l.numberFormat('#') + "p
+x";
                $(to + '_left').style.width = r.numberFormat('#') + "p
+x";

                var f = function () { poll_token(base_url,token,to, pw
+idth); };
                setTimeout(f,500);
            }
            else
            {
                if ( s.search(/^ok/i) > -1 )
                {

                    $(to + '_done').style.width = pwidth + "px";
                    $(to + '_done').style.background = 'green';
                    $(to + '_label').style.color = 'green';
                    $(to + '_label').innerHTML = "abgeschlossen ( 100%
+ )";

                    $(to + '_left').style.width = "0px";

                    $(to + '_ok').style.display = 'inline';
    
                    return true;
                }
                
                if ( s.search(/^error/i) > -1)
                {
                    $(to + '_done').style.background = 'red';
                    $(to + '_label').style.color = 'red';
                    
                    $(to + '_error').style.display = 'inline';
                    
                    return false;
                }
                
                var f = function () { poll_token(base_url,token,to, pw
+idth); };
                setTimeout(f,500);
            }
        }, 

        onFailure:function(t)
        {
            alert('Fehler beim Zugriff auf Webserver: ' + t.status + '
+ -- ' + t.statusText);
        }, 
        asynchronous:!check_sync
    });
}

[download]
And finally the server sided progress component 
package ECSInkassoWeb::Controller::Progress;

use strict;
use warnings;

use Data::Dumper;

use base qw(Catalyst::Controller ECSInkassoWeb::Controller::Base);

use FileSystemObjects::File;

sub get : Local
{
    my ( $self, $c ) = @_;
    
    my $token = $c->req->{parameters}->{token};

    my $konfig = ECS::Inkasso::Konfig->new($c);
    
    my $f = FileSystemObjects::File->new( $konfig->{location}->{progre
+ss} ."/$token" );
    my $h = $f->open("<");
    
    $_ = <$h>; chomp;
    
    $c->response->headers->header('content-type' => "text/plain; chars
+et=UTF-8");
    $c->response->body( $_ );
}

1;

[download]
The snippets are directly taken from the production server, and contain german. Let me now if that is problem.


holli, /regexed monk/

Replies are listed 'Best First'.
Re^2: Client Pull Not Working correctly
by Anonymous Monk on Aug 24, 2007 at 16:25 UTC
    Thanks. I could use some code from your example.
[reply]

In Section Seekers of Perl Wisdom