Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there.

I wrote a little program that inputs a list of files from an INI file then uses 'mirror' and 'getstore' from LWP::Simple to check if its updated then download the file.

So far this has worked fine but now my boss has asked for a file that is timestamped down to the minute.

Is there anyway to do a LS on a directory using LWP::Simple?

I need to either do a 'LS mib*.csv' or do an LS then use a Regular expression to find the filename.

Please Help.
Steve
cityscapeonline@@@yahoo...com...au

Replies are listed 'Best First'.
Re: LWP::Simple
by Zaxo (Archbishop) on Mar 26, 2002 at 01:27 UTC

    LWP::Simple's head() method gives a list whose third element is $modified_time. If you want to look at that when you download, you should use LWP::UserAgent and call $rsp->last_modified() of the HTTP::Response object returned.

    After Compline,
    Zaxo

[reply]
Re: LWP::Simple
by Dice (Beadle) on Mar 26, 2002 at 02:46 UTC

    I'd like to be able to trust headers, but I've been burned by this kind of thing too many times.

    Set up a cron job to get the file every minute, through LWP::Simple or anything else, really. (wget, whatever) Bring the file down under some name other than what it's real time, though. (E.g. append a "testing-" in front of it.) Then, "diff" it against a "baseline copy" of the most recently obtained version of the file of the same name. If the files differ, then your newly-obtained version can be used as the "current" file, and you can archive the old one. If the files are the same, delete the "testing-" version. Try again the next minute.

    Something to beware of: If you file really is changing on a minute to minute basis and you're trying to get this file via HTTP, then you will lose out on this file at some point in time. Servers go down, networks get hosed, things fall apart, the centre will not hold, etc. Please factor this into your plans.

    Cheers,
    Richard

[reply]
      The app already uses a cron to ftpget the files, the problem now is trying to get it to download a wildcard. Say there is a file on the server thats called hi20020326-1000.csv then hi20020326-1030.csv then hi20020326-1031.csv I need to say, get hi*.csv Thank you for you input anyways. Hope you can help with this
[reply]

        try using 'mget hi*.csv' where 'get hi*.csv' doesn't work.

        try reading the documentation for ftpget, i'm almost certain the answer to your question is there.

        check out Net::FTP and skip ftpget.

        tell the PHB to stop being so anal *or* hire a consultant to fix his data-management nightmare.

        Update: A google search turned up some circa 1993 pure perl/ftp code to get listings, files, and simple mirroring. I'm thinking about posting it for hy^Histe^Horical value, came in a email as a shar file no less...;)

        Any interest?

[reply]
Re: LWP::Simple
by RMGir (Prior) on Mar 26, 2002 at 01:10 UTC
    Hmm, what kind of server are you getting the files from, HTTP or FTP?

    If it's FTP, you can issue a command to request a dir; for HTTP, I think what you'd have to do is issue a request for the file if it's newer than the one you have.

    I'm pretty sure there's a header for that, but I can't remember offhand what it might be. Sorry!

    But look into HTTP headers on google or w3c.org, and you're likely to find what you're looking for.
    --
    Mike

[reply]

Back to Seekers of Perl Wisdom