RobTuck has asked for the wisdom of the Perl Monks concerning the following question:

Ive written a script to grab some files using LWP::Simple. The files are just plain text files which are part of a database for another site of mine.

The script seems to get the files fine, but im getting quite a few, and it rarely gets them all. Is there any way I can make the script wait until the file has definitely got the file before moving on?

Sorry if this is a stupid question, but ive not used the LWP::Simple module before.

Heres some of the code I have:

$sectionsurl = "http://www.url.com/data/faqdata/sections.toc";
$headerurl = "http://www.url.com/data/settings/header.txt";
$settingsurl = "http://www.url.com/data/settings/settings.txt";

# print out the settings area
$settingstext = get($settingsurl);
@settings = split(/\n/,$settingstext);

foreach $setting (@settings)
{
   print "[".$setting."]<br>";
}

# print out the header text
$headertext = get($headerurl);
$headertext =~ s/\n/\<br\>/ig;
print $headertext;

print "<br><br><hr><br><br>";

# print out the faq index

$allsections = get($sectionsurl);
@sections = split(/\n/,$allsections);

foreach $section (@sections)
{
   print "[".$section."]<br>";
}

[download]


Sometimes everything will display, and sometimes only one of the files will display. Id appreciate any advice.

Replies are listed 'Best First'.
Re: Randomly working script using LWP::Simple
by tcf22 (Priest) on Nov 28, 2003 at 00:53 UTC
    Check the return value and try grabbing the file a few times. 
    my $i = 0;
while(1){
  if($settingstext = get($settingsurl)){
    last;
  }else{
    if(++$i == 5){
      print "Can't get $settingsurl\n";
      ##More error handling stuff here
    }
  }
}

    [download]

    - Tom

[reply]
[d/l]
Re: Randomly working script using LWP::Simple
by Roger (Parson) on Nov 28, 2003 at 01:01 UTC
    You should check the returned value from get -
    $settingstext = get($settingsurl)
  or die "Can not fetch URL!";

    [download]
    What you could do is to repeat the get process until you succeed. You will probably need to implement some sort of retry timeout or number of retries to limit your loop.

    And also you should try to use the more capable LWP::UserAgent or WWW::Mechanize instead.

[reply]
[d/l]
      Er, no. LWP::Simple uses LWP::UserAgent. It just takes care of alot of the drudge-work. That said, getstore() is a more appropriate function than get().
[reply]
Re: Randomly working script using LWP::Simple
by RobTuck (Initiate) on Nov 28, 2003 at 01:21 UTC
    Thanks guys, the loop works perfectly. Hopefully it wont bog down when ive implemented it for the rest of the files.
[reply]

Back to Seekers of Perl Wisdom