zarath has asked for the wisdom of the Perl Monks concerning the following question:

Heyhey!

Perl-virgin here, be gentle, pretty please with sugar on top. + English is not my native tongue, but i pride myself in my capabilities (in written English anyways, I sound a bit like a mix of a Frenchman and a German trying to speak American).

I have a question about the very first script I'm trying to write.

I just took an existing script and with pretty much no experience with scripting tried to understand what was in there and now trying to re-write it with the help of the Google-Gods. Needless to say, I need some better (more specific/efficient) help. Please consider the following 'code':

#!perl -w # # Archiver for processed EDI messages # - move all messages to existing subfolder *\yyyy\mm\dd use strict; use warnings; use Time::Piece; use File::Find; use File::Copy; # no idea if I need all of these or not my $base = 'D:\Some\Specific\Folder\\'; my @time = localtime; my $mday = $time[3]; my $month = $time[4] + 1; $month = sprintf("%02d", $month); my $year = $time[5]+1900; my $from = $base.*\.'.txt'; # THE PROBLEM print ($from); # just a check to see what is used as $from, if anything # will obviously be left out when the rest works my $to = $base.$year."\\".$month."\\".$mday."\\"; # this folder already exists move($from,$to) or die $!; # might also be (part of) the problem print "moved ".$from." to ".$to; # a check to see what has been moved, if anything exit 0;

I know this lacks a bunch of things (like writing to a logfile), don't bother summing them up, I won't bother adding them anyway. Right now at least, might do when I get better at this stuff and have some left-over time on my hands. The only thing I need it to do on a daily basis is move a number of .txt-files (could be 500, could be 10,000 files) from folder 'D:\Some\Specific\Folder\' to the subfolder 'D:\Some\Specific\Folder\yyyy\mm\dd\' (obviously for today, this would be 'D:\Some\Specific\Folder\2017\05\23\'). As commented in the code, this subfolder will already exist when needed.

The big(gest) problem in this code is the attempt at using a wildcard, I just can not get my head around it. I know *\ will not work here, but I can't figure out what will. The *.txt files can have several different kinds of formatting. * always starts with a set code of 3 letters and numbers, but there are more than 10 different set codes; each set code has a different build for what comes between the set code and the '.txt'. Defining every one of these possibilities seems a bit redundant.

The script simply needs to check: Is this a .txt file? Yes: move to given subfolder! No: do not touch and move on! This looks like a quest easily achieved. But I have read some pieces about Perl that have made steam come out of my ears and stuff that has made me see the light (or so I thought until I tried it).

I have tried a large number of combinations with $ . / \ ' in my attempt at making "my $from = [...]" work - all based on stuff I have found via named websearch, none has cleared a path. Please don't make me sum them all up, I have no idea anymore, there have been too many attempts and I have not saved the losing versions apart from the one given above.

I'm sure this will be resolved in a jiffy by the first person who bothers to read this whole thing. Will you be that person?

Replies are listed 'Best First'.
Re: Wildcards for a wide array of strings?
by Eily (Monsignor) on May 23, 2017 at 08:22 UTC

    glob might help. It lets you find files that match a given pattern in a folder (it's not recursive). Something like: my @from = glob( quotemeta($base)."*.txt") where quotemeta escapes the special characters in $base, so that they are taken litteraly and not interpreted as pattern metacharacters.

    This returns a list of files though, so I stored it inside an array (@from) instead of a scalar ($from). You can then use a foreach loops to process the files individually

Re: Wildcards for a wide array of strings?
by Discipulus (Canon) on May 23, 2017 at 08:52 UTC
    Hello, zarath and welcome to the monastery and to the wonderful world of Perl,

    As (sigh) already said glob can help you if all files are currently on the same level, if not it will be worth to learn File::Find or how to searching a directory tree by hands. Many way are possible, as always in Perl, being my preferred the one shown by tachyon many years ago: Re: Win32 Recursive Directory Listing

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
Re: Wildcards for a wide array of strings?
by thanos1983 (Parson) on May 23, 2017 at 11:36 UTC

    Hellow zarath,

    Something like this should do what you want.

    #!/usr/bin/perl use strict; use warnings; use File::Copy; use Data::Dumper; use File::Path qw(make_path); my $dir = shift || "."; opendir(my $dh, $dir) or die "cannot open directory $dir $!"; my @files = grep { /\.txt$/ && -f "$dir/$_" } readdir($dh); closedir $dh; # 0 1 2 3 4 5 6 7 8 my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(t +ime); # $mon the month in the range 0..11 , with 0 indicating January and 11 $mon += 1; # $year contains the number of years since 1900. To get a 4-digit year + write: $year += 1900; # To get a 2-digit year write: my $short_year = substr $year, 2; my $new_dir = "$dir/$year/$mon/$mday"; eval { make_path($new_dir); 1 } or die "Can't create home directory: $@\n"; if (@files) { foreach my $file (@files) { # the perl file move function move("$dir/$file", $new_dir) or die "The move operation failed: $!"; } print Dumper \@files; } else { print "No files found to move\n"; } __DATA__ $ perl test.pl $VAR1 = [ 'counts.txt', 'file.txt', 'test.txt' ]; ~/Monks$ ll 2017/5/23/ total 16 drwxrwxr-x 2 tinyos tinyos 4096 May 23 13:29 ./ drwxrwxr-x 3 tinyos tinyos 4096 May 23 13:23 ../ -rw-rw-r-- 1 tinyos tinyos 0 May 9 15:19 counts.txt -rw-rw-r-- 1 tinyos tinyos 24 May 9 11:53 file.txt -rw-rw-r-- 1 tinyos tinyos 465 May 9 15:55 test.txt $ perl test.pl No files found to move

    If you want you can set permissions, read (File::Path). I also used (readdir and File::Copy).

    There are multiple ways of resolving your task, this is just one.

    Update:

    Adding if and else condition on greped files.

    Hope this helps.

    Seeking for Perl wisdom...on the process of learning...not there...yet!
Re: Wildcards for a wide array of strings?
by ww (Archbishop) on May 23, 2017 at 12:25 UTC

    For 'THE PROBLEM' you might want to consider using a loop, with a regex test for the files you want - a suggestion that infers from your text that the $from directory may contain non-.txt files.

    #!/usr/bin/perl -w use strict; # 1190936 my @sourcefiles = <DATA>; for $_(@sourcefiles) { if ($_ =~ /.+?\.txt/ ) { # process it here -- simulated by; print $_; } else { next; } } __DATA__ foo.bar bar.txt baz.bat blivitz.txt; nottext and somethingelse.txt

    Rewriting LN 5 to obtain data from your source directory is left as a (trivial) exercise -- once you do some reading, using perldoc open, for example.

    execution:

    C:\1190936.pl bar.txt blivitz.txt; somethingelse.txt
Re: Wildcards for a wide array of strings?
by Anonymous Monk on May 23, 2017 at 08:21 UTC
Re: Wildcards for a wide array of strings?
by zarath (Beadle) on May 23, 2017 at 14:02 UTC

    Thank you sooo much everyone! I finally got it up & running! Should have posted here earlier.

    I took items from pretty much everyone's advice, mixed them up and now it does what it's supposed to do.

    In case someone wants to know what it has eventually become and / or feels the need to make it more efficient:

    #!perl -w # # Archiver for processed EDI messages # - move all messages to existing subfolder *\yyyy\mm\dd use strict; use warnings; use Time::Piece; use File::Copy; use File::Path; use File::Glob; my $base = 'D:\\Some\\specific\\folder\\'; opendir (my $dhb, $base) or die "Cannot open ".$base.": ".$!; closedir $dhb; my @time = localtime; my $mday = $time[3]; my $month = $time[4] + 1; $month = sprintf("%02d", $month); my $year = $time[5]+1900; my $to = $base.$year."\\".$month."\\".$mday."\\"; opendir (my $dht, $to) or die "Cannot open ".$to.": ".$!; closedir $dht; my @files = glob($base."*"); foreach my $file (@files) { if ($file = glob(($base)."*.txt")){ move($file, $to) or die ("The move operation failed for ".$fil +e." because ".$!); print "Moved ".$file." to ".$to or die $!; } else { next; } } exit 0;

      Having gone to the effort of loading Time::Piece you may as well use it and avoid all those magic numbers.

      #!/usr/bin/env perl use strict; use warnings; use Time::Piece; my $to = localtime()->ymd ('\\'); print "$to\n";

      Hello again zarath,

      I noticed a few things on your code:

      You do not need use File::Path; and use File::Glob;.

      Also you do not need these lines, you do not do anything with the opendir(). You open and close the directory for no reason.

      my $base = 'D:\\Some\\specific\\folder\\'; opendir (my $dhb, $base) or die "Cannot open ".$base.": ".$!; closedir $dhb;

      I would prefer to get the time like this:

      # 0 1 2 3 4 5 6 7 8 my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(t +ime); # $mon the month in the range 0..11 , with 0 indicating January and 11 $mon += 1; # $year contains the number of years since 1900. To get a 4-digit year + write: $year += 1900;

      By doing so, you do not need the package use Time::Piece;.

      Update: I would also recommend to add and if() and else condition at your for loop. Also you can remove the else condition inside the loop in case the if() condition is not met it will jump straight to next;

      if (@files) { foreach my $file (@files) { if ($file = glob(($base)."*.txt")){ move($file, $to) or die ("The move operation failed for ".$fil +e." because ".$!); print "Moved ".$file." to ".$to or die $!; } next; } } else { print "No files found to move\n"; }

      These are some minor recommendations, hope you will love Perl and continue working on it.

      Seeking for Perl wisdom...on the process of learning...not there...yet!

      I used local paths and coding was done on a Unix-type system (MacOS in this case), so adjust paths and separators accordingly. Optionally, use the core module File::Spec and let it worry about the path separators. I present a few versions here.

      Each version adds an important check that you're not moving all your files one after the other into a single file (leaving just the contents of the last source file).

      In many cases you could skip using File::Copy and just use rename but then you may have to handle some edge cases yourself that the module handles for you. I'd recommend using the module.

      You definitely don't need to take a glob of all files, then in a loop glob all files ending in '.txt' again. You can use glob to get the list (in list context) of just the files you want and iterate over that. The code below does not do what you seem to expect.:

      if ($file = glob(($base)."*.txt")){ # ... }

      That is assigning the first match from glob(($base)."*.txt") to $file after you've already set $file as the loop variable. So for every file that matches glob($base/"*") you're re-globbing for "$base*.txt" using a fresh iterator internal to the perl runtime and operating on that filename that's returned in scalar context. Don't do that. Just get the right list the first time and operate on each member of that list in turn.

      Here's code with hard-coded paths in Unix path form to where I was working with the files.:

      #!/usr/bin/perl use strict; use warnings; use File::Copy (); my $base = '/Volumes/case-sensitive/projects/monks/1190936'; my @time = (localtime)[ 5, 4, 3 ]; $time[0] += 1900; $time[1]++; my $dir_for_today = sprintf '%s/%04d/%02d/%02d', $base, @time; die "destination $dir_for_today is not a directory!\n" unless -d $dir_ +for_today; foreach my $file ( glob qq($base/*.txt) ) { File::Copy::move $file, $dir_for_today or warn "Can't move $file + to $dir_for_today : $!"; }

      Here's equivalent code using File::Spec to worry about the path separator. :

      #!/usr/bin/perl use strict; use warnings; use File::Copy (); use File::Spec; my $volume = ''; # empty string for Unix my @dirs = ( File::Spec->rootdir, 'Volumes', 'case-sensitive', 'projec +ts', 'monks', '1190936' ); my $base = File::Spec->catpath( $volume, File::Spec->catdir( @dirs ), +'' ); my @time = (localtime)[ 5, 4, 3 ]; $time[0] += 1900; $time[1]++; $time[1] = sprintf '%02d', $time[1]; $time[2] = sprintf '%02d', $time[2]; my $dir_for_today = File::Spec->catdir( $base, @time ); die "destination $dir_for_today is not a directory!\n" unless -d $dir_ +for_today; foreach my $file ( glob File::Spec->catfile( $base, '*.txt' ) ) { File::Copy::move $file, $dir_for_today or warn "Can't move $file + to $dir_for_today : $!"; }

      For your path, according to your example I think you'd want 'D' for your volume, and of course you'd want 'Some', 'specific', 'folder' in your @dirs array according to your example.

      A couple more are below...

      So yeah, there are plenty of ways to accomplish this. This is but a pittance. Someone from here might suggest a modulino. Someone else may criticize using accessors in the OO version. Someone might suggest leaving the loop from the move method in the calling program. Software is flexible, programming languages are more so, and Perl more so than many programming languages. I hesitate to show it here, but this could also be done handily in a Perl (or Bash, or probably PowerShell) one-liner.

Re: Wildcards for a wide array of strings?
by zarath (Beadle) on May 24, 2017 at 07:59 UTC

    This is amazing, the more I learn about this stuff, the more I'm starting to like it. Started with a 'task' which I didn't really want to do (bit out of my comfort zone), but ended up wanting to learn more and more about this new world. Well, new to me obviously.

    Thank you very much for all the tips. I'll include the code I eventually ended up with. I think it's as short as it can possibly be and a lot of the suggestions are not included, this is because I like these things short, clean and clear (always try to do this with SQL too, with which I have more experience).

    Might include some kind of logging if it goes wrong at some point, but I can't see this doing weird stuff as the applications that use these folders do what they're supposed to do.

    #!perl -w # # Archiver for processed EDI messages # - move all messages to existing subfolder *\yyyy\mm\dd use strict; use warnings; use Time::Piece; use File::Copy; my $base = 'D:\\Some\\Specific\\Folder\\'; # The source folder my $to = $base.localtime()->ymd ('\\')."\\"; # The destination folder foreach my $file (glob qq($base\*.txt)) { File::Copy::move $file, $to or warn "Can't move $file to $to : $ +!"; } # The actual move exit 0; # The graceful ending
      Hello again zarath,

      glad you solved your issue and that you started to like the Perl many ways to problem solving.

      Perlmonks is a wonderful place to learn.

      Just a little consideration: finally you ended with 'D:\\Some\\Specific\\Folder\\' why?

      We all know MSwin32 is a weird platform and \\ will work, as will work c:/path\\to\\\another\\\\dir that is weird.

      Windows also accept the Linux directory separator / and I usaully use it in my programs: you can be interested by Paths in Perl and as the wise mr_mischief altready suggested, to use File::Spec module that with a bit of overhead makes your programs portable to different platform.

      L*

      There are no rules, there are no thumbs..
      Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

        Hi Discipulus,

        Deciding on the platform that we use is not up to me (above my paygrade). I think they landed on this back in the day because they needed something easily implemented and they needed it instantly. The company I work for kind of tried to ride a bicycle before it could crawl. Right now it's got walking down and learning to run. We'll have to look into other platforms, but at the moment it is not in the top 10 prio-list.