Aldebaran has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I've been rather ambitious with perl projects lately, porting my html template from a linux laptop to a windows 8.1 laptop with a cygwin installation. I was stunned with how little resistance I encountered, but I do have a couple problems. First, I'd like to address the main topic of this post, which is creating a sensible control for uploading binary content to my site. Let me show you what I have but preface it by saying that I'm without perltidy at the moment (see question 2).

sub upload { use strict; use warnings; use 5.010; use Net::FTP; use Data::Dumper; my ($rvars, $rftp) = @_; my %vars = %$rvars; # get local files into an array my @filetypes = qw/jpg jpeg png ogv mp4 m4v webm/; my $pattern = join '|', map "($_)", @filetypes; my @matching; opendir my $eh, $vars{"to_vids"} or warn "can't open vids $!\n"; while (defined ($_ = readdir($eh))){ next if m/~$/; next if -d; if ($_ =~ /($pattern)$/i) { push(@matching, $_); } } closedir $eh; # stat local files, create hash #my %stat = map { # lstat($_) or die "Can't lstat $_: $!"; # $_ => { # s => ( -s _ ), # } #} @matching; #my $hashref = \%stat; #print Dumper($hashref); $rftp->cwd("/pages/eh5v.files") or warn "cwd failed in main $!\n"; my $rlist = $rftp->ls(); say "list is @$rlist"; $rftp->binary or warn "binary failed$!\n"; for ( @matching) { #say "matching is @matching"; my $a = file($vars{"to_vids"},$_); my $sa = $a->stringify; #$rftp->put($sa, $_) or warn "put failed $@\n"; } $rftp->ls(); return $rftp; }

The relevant part of the output is:

loading vids list is . .. kitchen.jpg kitchen.m4v kitchen.webm stairs.jpg stairs.m4 +v

What am I trying to do? I want a control to loop through the above files, and first see whether it exists on the remote server. If not, we upload it. If yes, then I'm interested in whether they are either a differing size or creation date. That's what the commented lstat part was to address, but I got errors out of that, because I didn't build the full path. (I think.) If they are different, I want a prompt to overwrite with the default being yes. If they are not different, then I want a prompt to overwrite with the default being no. It would be cool if this could work on a five-second timer. Furthermore, I would like not to create race conditions. And finally, I'd like to have a portion of code to deal with error handling such as time-outs, or directory doesn't exist. The directory, of course "should" exist, but that's one of those words....

So that's very ambitious, and I'm not completely-wed to any part of the spec if it's ill-advised, but I thought I'd throw the whole thing out there in the original post. If I get it right once, it's eminently re-usable.

I have 2 other questions that are unrelated to the above. I don't think they deserve their own threads. To the extent that a subthread lasts more than one level, maybe we can rename it appropriately. How do I get CPAN to kickstart?

$ perl -MCPAN -e shell Set up gcc environment - 3.4.5 (mingw-vista special r3) cpan shell -- CPAN exploration and modules installation (v1.9800) Enter 'h' for help.

This hangs. I'm running the 64-bit cygwin download that is no more than 2 weeks old on a 64-bit windows 8.1 machine. Do I have cpan and not know it? I don't think so:

Fred@Psiborg ~/pages2/krov $ sudo cpan -bash: sudo: command not found Fred@Psiborg ~/pages2/krov $ cpan -bash: cpan: command not found Fred@Psiborg ~/pages2/krov $ cpan My :: Module -bash: cpan: command not found Fred@Psiborg ~/pages2/krov $

Finally, I reached the level of Scribe recently, and I think it would behoove me to have a better brand name. To this end, I'd like to change my username to something cooler and more-identifiable than my current one, which is just an arbitrary 'nym. How do I do that?

Replies are listed 'Best First'.
Re: a full-featured control for uploading files and a couple of other tidbits
by Athanasius (Archbishop) on May 23, 2015 at 07:50 UTC
Re: a full-featured control for uploading files
by Aldebaran (Curate) on May 24, 2015 at 00:25 UTC

    I realize I'm responding to my own original post. I want this to be the subthread to deal with the main control for uploads and have renamed it accordingly. I've got better results now, so I'd like to show newer code, output, and ask for code review to see if I am achieving an updated spec.

    sub upload { use strict; use warnings; use 5.010; use Net::FTP; use Path::Class; my ($rvars, $rftp) = @_; my %vars = %$rvars; # prelims with server...see if we're able to do anything $rftp->cwd("/pages/eh5v.files") or warn "cwd failed in main $!\n"; my $rlist = $rftp->ls(); say "remote list is @$rlist"; $rftp->binary or warn "binary failed$!\n"; # filter for filetypes my @filetypes = qw/jpg jpeg png ogv mp4 m4v webm/; my $pattern = join '|', map "($_)", @filetypes; opendir my $eh, $vars{"to_vids"} or warn "can't open vids $!\n"; while (defined ($_ = readdir($eh))){ next if m/~$/; next if -d; next unless m/($pattern)/i; my $full_path = file($vars{"to_vids"}, $_); my $string_path = $full_path->stringify; my $size_local = -s $string_path; say "value for $_ is $size_local"; #find size on server my $size_remote = $rftp->size($_) or warn "size query failed $! fo +r $_\n"; say "size remote for $_ is $size_remote"; ## control for upload cases if (!defined($size_remote)) { say "Upload $_" }; if ($size_remote eq $size_local) { say "Sizes equal with $_. Overwrite? Default=n" } else { say "Sizes different with $_. Overwrite? Default=y" }; } closedir $eh; return $rftp; }
    size query failed for stairs.webm Use of uninitialized value $size_remote in concatenation (.) or string + at template_stuff/html2.pm line 331. Use of uninitialized value $size_remote in string eq at template_stuff +/html2.pm line 336. loading vids remote list is . .. kitchen.jpg kitchen.m4v kitchen.webm stairs.jpg st +airs.m4v value for kitchen.jpg is 27261 size remote for kitchen.jpg is 27261 Sizes equal with kitchen.jpg. Overwrite? Default=n value for kitchen.m4v is 19297258 size remote for kitchen.m4v is 19297258 Sizes equal with kitchen.m4v. Overwrite? Default=n value for kitchen.webm is 12808479 size remote for kitchen.webm is 12808479 Sizes equal with kitchen.webm. Overwrite? Default=n value for stairs.jpg is 99199 size remote for stairs.jpg is 99199 Sizes equal with stairs.jpg. Overwrite? Default=n value for stairs.m4v is 41678290 size remote for stairs.m4v is 40860741 Sizes different with stairs.m4v. Overwrite? Default=y value for stairs.webm is 30127462 size remote for stairs.webm is Upload stairs.webm Sizes different with stairs.webm. Overwrite? Default=y

    I'm really pleased with this early result. Instead of testing whether a file exists on the remote server, I simply query for the size and if I get undef, I take that to indicate non-existence, as the above output confirms. Also, just looking at these data makes me believe that size really tells the story of the revision history (how do you alter video without changing the size 99.9% of the time?), so I'll re-state the objective with this in mind:

    I want a control to loop through binary files, and first see whether it exists on the remote server. If not, we upload it. If yes, then I'm interested in whether they are a differing size. If they are different, I want a prompt to overwrite with the default being yes. If they are not different, then I want a prompt to overwrite with the default being no. It would be cool if this could work on a five-second timer. Furthermore, I would like not to create race conditions. And finally, I'd like to have a portion of code to deal with error handling such as time-outs, or directory doesn't exist.

    I made a lot of mistakes with stat and lstat:

    my (undef,undef,undef,undef,undef,undef,undef,$size, $atime,$mtime,$ctime,undef,undef) = lstat($string_path); say "File $string_path has $size bytes $mtime"; my $born_local = stat($string_path)->ctime ); my $size_local = stat($string_path)->size ); say "values for $_ are $born_local and $size_local";

    I arrived at the opinion that using the native perl syntax was not only simpler, but for me, darn near necessary. This might be halfway there, so I'd appreciate any tips/criticisms. Otherwise, I'll just continue. Thanks for your attention.

      Hello again, Datz_cozee75,

      For user input with timeout, you can use the Prompt::Timeout module:

      use Prompt::Timeout; use constant TIMEOUT => 5; ... if (defined($size_remote)) { my $default = ($size_remote == $size_local) ? 'N' : 'Y'; my $answer = prompt('Overwrite?', $default, TIMEOUT); upload($_) if $answer =~ /^Y/i; # overwrite } else { upload($_); }

      Notes:

      • If $size_remote is undef, testing it with eq or == will produce a warning — so don’t do that.

      • $size_remove and $size_local are both integers, so prefer == to eq for the equality comparison.

      Two additional observations:

      1. The loop condition:

        while (defined ($_ = readdir($eh)))

        can be written more concisely as:

        while (readdir($eh))
      2. It’s inefficient to use captures in a regex if those captures aren’t needed. Use grouping (non-capturing) parentheses:

        my $pattern = join '|', map "(?:$_)", @filetypes;

        or, in this case, omit the parentheses altogether:

        my $pattern = join '|', @filetypes; ... while (readdir($eh)) { next if /~$/ || -d || !/$pattern/i; ...

      Hope that helps,

      Athanasius <°(((><contra mundum Iustus alius egestas vitae, eros Piratica,

        Thanks so much, Athanasius, this is shaping up really nicely now. The queries are cadenced well and have a nice aesthetic about them as they all align, much like a cpan installation. This is typical output:

        ... value for stairs.jpg is 99199 size remote for stairs.jpg is 99199 Overwrite? [N] :N value for stairs.webm is 30127462 size query undef for stairs.webm stairs.webm doesn't exist 1 uploading stairs.webm ftp return was stairs.webm ...

        I couldn't understand for a while why the first binary was alway being re-uploaded, so I turned up the debug level on Net::FTP and found my answer:

        value for kitchen.m4v is 19297258 Net::FTP=GLOB(0x98127b4)>>> HELP SIZE Net::FTP=GLOB(0x98127b4)<<< 214 Syntax: SIZE <sp> pathname Net::FTP=GLOB(0x98127b4)>>> SIZE kitchen.m4v Net::FTP=GLOB(0x98127b4)<<< 550 SIZE not allowed in ASCII mode size query undef for kitchen.m4v kitchen.m4v doesn't exist Net::FTP=GLOB(0x98127b4)>>> CWD /pages/eh5v.files Net::FTP=GLOB(0x98127b4)<<< 250 CWD command successful Net::FTP=GLOB(0x98127b4)>>> TYPE I Net::FTP=GLOB(0x98127b4)<<< 200 Type set to I 1 uploading kitchen.m4v

        Here's what I have now, with upload() doing 4 different batches of ftp'ing of differing sets of files to different places.

        sub upload { use strict; use warnings; use 5.010; use Net::FTP; use Path::Class; use Prompt::Timeout; use constant TIMEOUT => 5; my ( $rvars, $rftp ) = @_; my %vars = %$rvars; # prelims for server...see if we're able to do anything my $target = "/pages/eh5v.files"; $rftp->binary or die "binary failed$!\n"; #necessary for size req +uest $rftp->cwd($target) or warn "cwd failed in main $!\n"; my $rlist = $rftp->ls(); say "remote list is @$rlist"; # filter binaries for filetypes my @filetypes = qw/jpg jpeg png ogv mp4 m4v webm/; my $pattern = join '|', @filetypes; opendir my $eh, $vars{"to_vids"} or warn "can't open vids $!\n"; while ( readdir($eh) ) { next if /~$/ || -d || !/$pattern/i; my $full_path = file( $vars{"to_vids"}, $_ ); my $string_path = $full_path->stringify; my $size_local = -s $string_path; say "value for $_ is $size_local"; #find size on server my $size_remote = $rftp->size($_) or warn "size query undef for $_ +\n"; if ( defined($size_remote) ) { say "size remote for $_ is $size_ +remote" } else { say "$_ doesn't exist" } ## control for upload cases if ( defined($size_remote) ) { my $default = ( $size_remote == $size_local ) ? 'N' : 'Y'; my $answer = prompt( 'Overwrite?', $default, TIMEOUT ); ftp_binary( $string_path, $_, $rftp, $target ) if $answer =~ /^Y/i; # overwrite } else { ftp_binary( $string_path, $_, $rftp, $target ); } } closedir $eh; # make new directory for page's specific images my $path5 = dir( "", "images", $vars{remote_dir} ); my $string_path5 = $path5->stringify; say "string path5 is $string_path5"; my $mk_return = $rftp->mkdir($string_path5) or warn "mkdir failed in upload $!\n"; say "mk_return is $mk_return"; ######### this part is clunky my $refc = $vars{"ref_content"}; my @AoA = @$refc; for my $i ( 0 .. $#AoA ) { my $a = file( $vars{to_images}, $AoA[$i][0] ); my $sa = $a->stringify; my $b = file( $AoA[$i][0] ); my $sb = $b->stringify; ftp_binary( $sa, $sb, $rftp, $string_path5 ); } #load css file to server my $path3 = file( $vars{"template_path"}, $vars{"css_file"} ); my $string_path3 = $path3->stringify; my $css_return = ftp_ascii( $string_path3, $vars{"css_file"}, $rftp, + "/css" ) or warn "put failed $@\n"; say "css return was $css_return"; #load html file to server my $path4 = file( $vars{"base_path"}, $vars{"html_file"} ); my $string_html = $path4->stringify; my $html_return = ftp_ascii( $string_html, $vars{"html_file"}, $rftp, "/pages" ) or warn "put failed $@\n"; return $html_return; }

        And here are the routines that do the actual ftp'ing:

        sub ftp_binary { use strict; use warnings; use 5.010; use Net::FTP; my ( $from, $to, $rftp, $whereto ) = @_; my $cwd_return = $rftp->cwd($whereto) or die "cwd failed to $whereto + $!\n"; $rftp->binary or die "binary failed$!\n"; say "$cwd_return uploading $to"; my $ftp_return = $rftp->put( $from, $to ) or warn "put failed for $t +o $@\n"; say "ftp return was $ftp_return"; return $ftp_return; } sub ftp_ascii { use strict; use warnings; use 5.010; use Net::FTP; my ( $from, $to, $rftp, $whereto ) = @_; my $cwd_return = $rftp->cwd($whereto) or die "cwd failed to $whereto + $!\n"; $rftp->ascii or die "ascii failed$!\n"; say "$cwd_return uploading $to"; my $ftp_return = $rftp->put( $from, $to ) or warn "put failed for $t +o $@\n"; say "ftp return was $ftp_return"; return $ftp_return; }

        This works, in that it works under best conditions, so I'm still fishing for what I would do if, say, the server didn't respond for a while. I'll be able to post return values when that does happen, and it happens a lot, for whatever reason. I wanted to get this posted for code review. I appreciate any suggestions. Thanks again, Athanasius.