Downloading only new files added to a local path with Perl (FTP)

SaraMirabi has asked for the wisdom of the Perl Monks concerning the following question:

Hello Dears,

I want to download all files that have been loaded to a directory with FTP automatically with the below code. the code can run but with below error:

Can't call method "mtime" on an undefined value at -e line 1.

Also, however it gets above error it downloads all the files in the path, not only added newly added files. I really appreciate if you help me.

I want to schedule this Perl code to be run at a specific time, so every day I don't need to get the files manually.

#!/bin/bash


use strict;
use warnings;
use diagnostics;

# Login credentials

user='*****' #Do not forget to enclose inside single or double quotes

pass='*****'

directory='Remoredirectory'

host='RemoteIP'



# Set time

# perl -e 'print time' will return the current time in Unix epoch form
+at

# So if you substract 43200, that should give you the current time min
+us 30 minutes in Unix epoch format:

time=$(perl -e 'print time-43200')



# Connect to host and download the listing of the remote directory

ftp -n $host <<END_GET_LIST

        quote USER $user

        quote PASS $pass

        cd $directory

        ls -l /Localdirectory/ftpList.txt

        quit

END_GET_LIST

# Disconnect from remote host



# Save the 6th, 7th, and 8th field of the directory listing (i.e. Aug 
+15 5:15) of each line into file Dates.txt

awk -F ' ' '{print $6,$7,$8}'/Localdirectory/ftpList.txt > /Localdirec
+tory/'Dates.txt'



# Save the 9th field of the directory listing (file name) of each line
+ into file Files.txt

awk -F ' ' '{print $9}' /Localdirectory/ftpList.txt > /Localdirectory/
+'Files.txt'



linenum=0 #Auxiliary variable for sed.



while read list; do

        linenum=$((linenum+1))

        #Convert the modification datetime of each file to Unix's epoc
+h

        epoch=$(perl -MFile::stat -e "print stat('$list')->mtime")

        if [ $epoch -gt $time ] ; then

                file=$(sed -n "${linenum}p" Files.txt)  #If the condit
+ion is satisfied, use sed to get the name of the file

                 #in the same line of Files.txt.

#Connect again and download file when the condition above has been sat
+isfied.

                ftp -n $host <<END_RETRIEVE

                        quote USER $user

                        quote PASS $pass

                        cd $directory

                        binary
                        
                        get $file

                        quit

END_RETRIEVE

        fi

done < /Localdirectory/'Dates.txt'
[download]

Comment on Downloading only new files added to a local path with Perl (FTP) Download Code

Replies are listed 'Best First'.
Re: Downloading only new files added to a local path with Perl (FTP) by haukex (Archbishop) on Aug 19, 2020 at 13:17 UTC
`epoch=$(perl -MFile::stat -e "print stat('$list')->mtime")` I would guess that `$list` contains special characters that are preventing correct interpolation. Try: `epoch=$(perl -MFile::stat -se 'print stat($list)->mtime' -- -list="$li +st") # or, better: epoch=$(perl -MFile::stat -wMstrict '-Mvars=$list' -se 'print stat($li +st)->mtime' -- -list="$list")` [download] See perlrun for the `-s` switch. Of course, the very best way is to rewrite your entire bash script as a Perl script. And why does your bash script start with `use strict;`? For bash scripts, see Safer bash scripts with `set -euxo pipefail`.	[reply] [d/l] [select]
Re: Downloading only new files added to a local path with Perl (FTP) by salva (Canon) on Aug 19, 2020 at 13:46 UTC
There are already plenty of utilities that can do that while handling errors an corner cases correctly. For instance, I am a big fan of lftp, its `mirror` command can do what you are asking for.	[reply] [d/l]
Re: Downloading only new files added to a local path with Perl (FTP) by roboticus (Chancellor) on Aug 19, 2020 at 13:26 UTC
SaraMirabi: I'm not 100% certain, but the problem looks like you're not properly interpolating the bash variable list into line 72: `epoch=$(perl -MFile::stat -e "print stat('$list')->mtime")` So perl is likely trying to execute: `perl -MFile::stat -e "print stat('')->mtime"` Since you don't have a file named '' in your local filesystem, the stat function returns undef, which is why you get the "Can't call method "mtime" on an undefined value at -e line 1" error. If you put a "set -x" and "set -v" commands in your bash script, you'll be able to see what the expanded command lines expand to in your script, so you can more easily see what bash is telling perl what to do for you. Also, while I'm happy you're trying to "use strict" and "use warnings", I don't think they'll work well in bash. ;^) ...roboticus When your only tool is a hammer, all problems look like your thumb.	[reply] [d/l] [select]
Re: Downloading only new files added to a local path with Perl (FTP) by AnomalousMonk (Archbishop) on Aug 19, 2020 at 17:17 UTC
As an aside: `# perl -e 'print time' will return the current time in Unix epoch form +at # So if you substract 43200, that should give you the current time min +us 30 minutes in Unix epoch format: time=$(perl -e 'print time-43200')` [download] There are 86400 seconds in a day, so subtracting 43200 subtracts half a day. Subtract 1800 to go back half an hour. `c:\@Work\Perl\monks>perl -wMstrict -le "my $t = time; print $t; print scalar localtime $t; print scalar localtime $t - 43200; print scalar localtime $t - 1800; " 1597857155 Wed Aug 19 13:12:35 2020 Wed Aug 19 01:12:35 2020 Wed Aug 19 12:42:35 2020` [download] See time, localtime, gmtime. Give a man a fish: `<%-{-{-{-<`	[reply] [d/l] [select]
Re: Downloading only new files added to a local path with Perl (FTP) by jcb (Parson) on Aug 20, 2020 at 01:12 UTC
I am not entirely sure how best to say this... but that is a shell script. The `use strict;` and so on will only produce errors from the shell about not having a command named "use". The first step in doing this with Perl would probably be to look into modules like Net::FTP and LWP::UserAgent that handle FTP.	[reply] [d/l]
Re: Downloading only new files added to a local path with Perl (FTP) by perlfan (Parson) on Aug 19, 2020 at 17:02 UTC
If you have `ssh` access for `rsync`, this would work better. `wget` is also a good tool for this.	[reply] [d/l] [select]