b_gsmls has asked for the wisdom of the Perl Monks concerning the following question:

I've been searching around, but am not having any luck. I'm reading in a file into an array and looping line by line. It works great for the first ~4142 lines, then breaks. If i remove those first 4142 lines, rerun program, i can get through about 13k more lines then breaks again. Removing those lines from the file, the program immediately breaks at and error is thrown for the last line in file.
Any wisdom?
File Ex:
/imagedb/highres/01/1540901_4.jpg
/imagedb/highres/01/1540901_5.jpg
/imagedb/highres/01/1543001.jpg
/imagedb/highres/01/1540001.jpg
error: Use of uninitialized value in concatenation (.) or string at ./thumb_test2.pl line 54, <MYINPUTFILE> line 60608
... open(MYINPUTFILE, "<$load_file"); # open for input my(@lines) = <MYINPUTFILE>; # read file into list my($line); foreach $line (@lines) # loop thru list { chomp $line; @h = split(/\//, $line); #split the "/" to get hash value for dir my $hash = "$h[3]"; my $JPG = "$h[4]"; my $outfile = "/imagedb/thumb/$hash/$JPG"; #out put image my $out_path = "/imagedb/thumb/$hash"; my $image = Image::Resize->new($line); #make image my $gd = $image->resize(160, 120); open(FH, ">$outfile"); #print the image file print FH $gd->jpeg(); close(FH); #close image } close(MYINPUTFILE); ....

Replies are listed 'Best First'.
Re: Reading 60k lines in file
by ikegami (Patriarch) on Oct 08, 2009 at 16:24 UTC

    It works great for the first ~4142 lines, then breaks.

    What does "breaks" mean exactly? Are you referring to the emission of the warning you mentioned?

    error: Use of uninitialized value in concatenation

    That's a non-fatal error (warning).

    You didn't indicate which line is line 54, but it's gotta be because $h[3] or $h[4] don't contain what you expect them to be. Some line of your input doesn't have any many slashes as you expect.

    Because you read the entire file at once (why?!), the line number in the error message (60608) is wrong. Fix your code as follows and you'll get the line number of your input that's incorrect. And you'll use must less memory too!

    open(my $in_fh, '<', $load_file) or die("Can't open input file \"$load_file\": $!\n"); while (my $in_qfn = <$in_fh>) { chomp($in_qfn); my ($rel_path) = $in_qfn =~ m{/([^/]+/[^/]+)\z} or do { warn("Line $. is malformatted. Skipping\n"); next; }; my $image = Image::Resize->new($_); my $gd = $image->resize(160, 120); my $out_qfn = "/imagedb/thumb/$rel_path"; open(my $out_fh, ">", $out_qfn) or do { warn("Can't create file \"$out_qfn\": $!. Skipping\n"); next; }; binmode($out_fh); if (!( print($out_fh $gd->jpeg()) and close($out_fh) )) { warn("Error writing file \"$out_qfn\": $!. Skipping\n"); close($out_fh); unlink($out_qfn); next; } }

    I added some much-needed I/O error checking.

    Update: Fixed extraneous semicolon.

      i'm giving this a whirl ,thanks much ikegami, I obviously could have written it better from the start. I am a bit unclear on the if statement so I'll need to do some reading..
        if (!( print($out_fh $gd->jpeg()) and close($out_fh) )) { warn("Error writing file \"$out_qfn\": $!. Skipping\n"); close($out_fh); unlink($out_qfn); next; }
        means
        if (!print($out_fh $gd->jpeg())) { warn("Error writing file \"$out_qfn\": $!. Skipping\n"); close($out_fh); unlink($out_qfn); next; } if (!close($out_fh)) { warn("Error writing file \"$out_qfn\": $!. Skipping\n"); close($out_fh); # Useless unlink($out_qfn); next; }
Re: Reading 60k lines in file
by jakobi (Pilgrim) on Oct 08, 2009 at 16:14 UTC

    Also add checks for your implied assumptions: does the split succeed and return 5 elements; what's with the open and the 2 gd operations in case of errors?

    An example: for the split, you could use a more readable version of e.g. @h=split(/.../, $line) and $#h>=4 or do {warn "illegal line: $line"; next} to bail out on questionable input.

    Also of interest for you: is it always the same line / file creating the problem? Missing or bad image file, ....? Which may make one of the gd calls die. Test returns before continue to use them. Try wrapping in eval{} and then testing $@ (e.g. do{warn ...; next} if $@)on calls that can die and warn/skip for the problem line.

    HTH, Peter
      The elements are confirmed to return properly, it works for several thousand images.
      It's always the same line, yes. I have 100 of these files, 33k -60k lines. each one works up to ~4120, breaks. then ~15k more work with no error, then break with error code
        The elements are confirmed to return properly

        Not in your provided code.

        it works for several thousand images

        That translates to 'infrequent error', and to 'can process some files successfully'. As do the statements about another 15K ... .

        It's always the same line, yes.

        Same line in the code, or hopefully and VERY helpful in debugging: always the same $line in the INPUT!? (Update: is it always the same image file or not? See Ikegamis comment below as well! A warn "input: $line\n" is a bit verbose but will pinpoint the file in question)

        Please check this and do add code to check the returns esp. from split and gd.


        Update: So you say, it's not the same file triggering the error. Not so nice. Ok. Try Ikegami's code or update your own suitably. If things still fail, then change

        my $gd = $image->resize(160, 120);
        my $gd; eval{$gd=$image->resize(160, 120)}; do{warn "processing error for file: $line - skipping\n"; next} if $@ o +r not ref $gd;

        Similar caution for the 2nd gd call. I'm assuming you've some error + sanity checking by now for the open, split and gd calls.

Re: Reading 60k lines in file
by wfsp (Abbot) on Oct 08, 2009 at 15:53 UTC
    Try
    while (my $line = <MYINPUTFILE>){ ... }
    foreach $line (@lines){ ... }
    is probably trying to read the whole file into @lines and _then_ loop over it. The while will read one line at a time.

    Update
    Better, check that $h[3] and $h[4] actually contain what you think they do.

      They do. I was printing them out to verify. The program does work for several thousand lines, then breaks at Image::Resize->new($line). which is line 54
      04
      1219504.jpg
      /imagedb/highres/04/1219504.jpg
      /imagedb/thumb/04/1219504.jpg
      /imagedb/thumb/04
Re: Reading 60k lines in file
by kennethk (Abbot) on Oct 08, 2009 at 16:02 UTC
    If you want to avoid the error you've quoted, you need to avoid using an uninitialized value in concatenation (.) or string at ./thumb_test2.pl line 54. Of course, I have no idea which line is 54, but I'll bet it's one of the many of the form my $hash = "$h[3]";. Assuming you are elevating warnings to errors, this will throw that error in cases where $h[3] is uninitialized, a.k.a. if line only had 3 elements for the example case. You could fix this by replacing that with

    my $hash = defined $h[3] ? "$h[3]" : q{};

    which will interpolate if the value is defined and will store an empty string (not an undef) in your variables.

      actually, the error points to the variable $line , the directory. going to try the while suggestion ...
Re: Reading 60k lines in file
by vitoco (Hermit) on Oct 08, 2009 at 20:40 UTC

    Is there a good reason to read and keep all that lines in an array? I don't see a sort or such...

    Try the following:

    ... open(MYINPUTFILE, "<$load_file"); # open for input while (my $line = <MYINPUTFILE>) # loop thru list { chomp $line; my ($hash, $JPG) = ($line =~ m!/([^/]+)/([^/]+)$!) or die; ...

    Update: restored $line var into this code.