in reply to Help with LWP::Simple

I'd like to be able to trust headers, but I've been burned by this kind of thing too many times.

Set up a cron job to get the file every minute, through LWP::Simple or anything else, really. (wget, whatever) Bring the file down under some name other than what it's real time, though. (E.g. append a "testing-" in front of it.) Then, "diff" it against a "baseline copy" of the most recently obtained version of the file of the same name. If the files differ, then your newly-obtained version can be used as the "current" file, and you can archive the old one. If the files are the same, delete the "testing-" version. Try again the next minute.

Something to beware of: If you file really is changing on a minute to minute basis and you're trying to get this file via HTTP, then you will lose out on this file at some point in time. Servers go down, networks get hosed, things fall apart, the centre will not hold, etc. Please factor this into your plans.

Cheers,
Richard

Replies are listed 'Best First'.
Re: Re: LWP::Simple
by Anonymous Monk on Mar 26, 2002 at 05:50 UTC
    The app already uses a cron to ftpget the files, the problem now is trying to get it to download a wildcard. Say there is a file on the server thats called hi20020326-1000.csv then hi20020326-1030.csv then hi20020326-1031.csv I need to say, get hi*.csv Thank you for you input anyways. Hope you can help with this
[reply]

      try using 'mget hi*.csv' where 'get hi*.csv' doesn't work.

      try reading the documentation for ftpget, i'm almost certain the answer to your question is there.

      check out Net::FTP and skip ftpget.

      tell the PHB to stop being so anal *or* hire a consultant to fix his data-management nightmare.

      Update: A google search turned up some circa 1993 pure perl/ftp code to get listings, files, and simple mirroring. I'm thinking about posting it for hy^Histe^Horical value, came in a email as a shar file no less...;)

      Any interest?

[reply]

In Section Seekers of Perl Wisdom