King0 has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to put together code that will find all the files within a directory, search for a particular string within each file and replace it accordingly. Eventually, I'd like to make the code look recursively through directories. Below is the code I've put together; there are two problems with it:

  1. It doesn't get all the files in the directory
  2. It doesn't search through each file making the desired replacement

I'd appreciate help in understanding where I went wrong.

use warnings; use strict; my $curUrl = "http://localhost:8080"; my $tarUrl = "http://localhost"; opendir DIR, $ARGV[0] or die "Could not open directory"; my @files = grep { -f $_ } readdir DIR; foreach (@files) { open FILE, "+>", "$ARGV[0]\\$_" or die "Could not open file -- $_" +; while (my $line = <FILE>) { $line =~ s/$curUrl/$tarUrl/g; } close FILE; } closedir DIR;

Replies are listed 'Best First'.
Re: Searching and Replacing file content within directory
by moritz (Cardinal) on Nov 12, 2010 at 14:10 UTC
    It doesn't get all the files in the directory

    Does the readdir call omit some files? Or are the files processed, and you just see no effect? (Data::Dumper and a few print statements can help you debugging) Is there some pattern in the names of missing files?

    For recursive traversal, use File::Find.

Re: Searching and Replacing file content within directory
by Tanktalus (Canon) on Nov 12, 2010 at 15:21 UTC

    Even on Windows, you can use / instead of \ for path separators. Makes things juts a bit more readable.

    That said, you should not be modifying files in-place. It's dangerous and not easy to do right. (Tie::File might make it doable.)

    Normal behaviour is: a) rename file to $_.old; b) open $_.old for read and $_ for write; c) loop through read-handle : modify line, write line; d) close filehandles; e) possibly delete old file (if you don't, then deleting it before (a) is a good thing, just in case it's still around).

    Note that as far as I can see, you're not writing anything back to the file, which, as I said, would be difficult to get correct anyway, so don't do it. But I wouldn't expect ANY files to be updated with your code.

      and just for completeness .. and becuase I, have made this oversight ...

      c.1) : read AND write the remaining lines from the input file. d) Then close the filehandles

      I expect that most otherfolk are more attnetive than this most of the time. BUT using myslef as a bench mark ... better cautionary than assumptive.

      Misha/Michael - Russian student, grognard, bemused observer of humanity and self professed programmer with delusions of relevance
Re: Searching and Replacing file content within directory
by fisher (Priest) on Nov 12, 2010 at 14:27 UTC
    need a recursion? well then do it:
    #!/usr/bin/env perl use warnings; use strict; my $curUrl = "http://localhost:8080"; my $tarUrl = "http://localhost"; my @buf; my ($line, $f); sub diveinto { print "dive into: $_[0]\n"; opendir DIR, $_[0] or die "Could not open directory"; my @files = readdir DIR; closedir DIR; foreach $f (@files) { if ($f =~ /\.{1,2}/) {next } print "processing: $f\n"; if ( -d "$_[0]/$f" ) { diveinto ("$_[0]/$f") } else { open FILE, "<". "$_[0]/$f" or die "Could not open file -- +$_[0]/$f"; @buf = <FILE>; close FILE; open W, ">" . "$_[0]/$f" or die "cunt open for r/w -- $_[0 +]/$f"; foreach $line (@buf) { $line =~ s/$curUrl/$tarUrl/g and print "Yes"; print W $line; } close W; } } } diveinto $ARGV[0];
    * I leave debug 'print'-s as is to help you understand what it does. It can be shrinked twice in size.
    * If it is a directory, we just call ourselves recursively
    * The original loop where you trying to replace string need to write result of s///;
Re: Searching and Replacing file content within directory
by chrestomanci (Priest) on Nov 12, 2010 at 15:02 UTC

    I think a better way to recursively process all files would be to pass your processing subroutine as a code reference to File::Find.

    use warnings; use strict; use File::Find my $curUrl = "http://localhost:8080"; my $tarUrl = "http://localhost"; sub process_files { next unless -f $File::Find::name; print "Processing $File::Find::name\n"; open FILE, "+>", $File::Find::name or die "Could not open file $File::Find::name"; while (my $line = <FILE>) { $line =~ s/$curUrl/$tarUrl/g; } close FILE; } find( &process_files, $ARGV[0] );

    It is a bit old school perl, but sometimes I find the old tools are the best.

      I too would advocate using the File::Find approach - only issue I can see with the solution from chrestomanci is that it doesn't update the files themselves.

      To emulate what perl's -i switch does, first rename each file with an extension like .orig, open that new file for reading, and write output to the original filename.

Re: Searching and Replacing file content within directory
by 7stud (Deacon) on Nov 12, 2010 at 21:18 UTC

    I'd appreciate help in understanding where I went wrong.

    This code:

    while (my $line = <FILE>) { $line =~ s/$curUrl/$tarUrl/g; }

    says to insert each line of FILE into the variable $line, then perform a substitution on the $line variable. Notably, you never printed $line to a file, so it can't end up in any file. The only thing you changed was the $line variable.

    Some preliminary things first: you should NOT use bareword file handles. The modern way to open a file is like this:

    open my $INFILE, '<', 'some_file' or die "Couldn't open $fname: $!";

    You should use a variable instead of a bareword filehandle, and you should include $! in the error message, which contains an explanation of what went wrong.

    You need to realize first and foremost that generally you CANNOT do inplace editing of a file. You have to read the old file line by line and write each line to a new file. You alter the lines you want before printing them to the new file. Then you can delete the old file, unlink(), and rename the new file to the old file name, rename(). You should try doing that by hand before employing some of perl's tricks, which allow you to mimic that process behind the scenes.

    In any case, the -i command line switch trick, which mimics inplace editing, can be duplicated inside a perl script by assiging a chosen backup extension to perls "in place editing" global variable $^I. Thereafter, STDOUT will be redirected to the new file, i.e. print()'s will go to the new file, and perl will handle the deleting and renaming behind the scenes.

    $^I (or $INPLACE_EDIT) tells perl that files processed by the <> construct are to be edited in-place. It's the same as the -i command line switch.

    http://www.tek-tips.com/faqs.cfm?fid=6549

    The code for doing that will probably confuse you more than doing it by hand yourself (untested):

    { local $^I = ".backup"; @ARGV = @files; while (<>) { s/$target/$replacement/g; print; } }

    ...or better:

    ... use English '-no_match_vars'; ... { local $INPLACE_EDIT = ".backup"; @ARGV = @files; while (<>) { s/$target/$replacement/g; print; } }

    Note that the code above creates files with the extension .backup for every file. I cannot find any information about what you are supposed to assign to $^I if you don't want backup files--maybe a blank string? But I read that Windows won't let you use $^I without creating backup files anyway.

    I think the value of $^I works like this:

    $^I = undef; #default value, Turn off inplace editing mode. $^I = "some_string"; #Turn on inplace editing mode, use "some_string" #as the extension for the backup files. $^I = ""; #Turn on inplace editing mode, delete backup files +.

      > I cannot find any information about what you are supposed to assign to $^I if you don't want backup files--maybe a blank string?

      Your guess is correct - setting $^I to an empty string overwrites the original file, no backups.


      --
      "Language shapes the way we think, and determines what we can think about."
      -- B. L. Whorf

        Thanks for confirming that. Just one correction: there's never any "overwriting" of the original file--a new file is always created. It's just that when you assign a blank string to $^I, perl doesn't save a copy of the original file before renaming the new file to the original file name.

Re: Searching and Replacing file content within directory
by zentara (Cardinal) on Nov 12, 2010 at 18:31 UTC
    This is something I've been using: It may have a few bugs, like do you want to search thru binary files too? There are many details to consider, like maintaining file permissions.
    #!/usr/bin/perl # Recursively searchs down thru directories replacing patterns in file +s. # usage zsr 'search' 'replace' 'ext'(optional with no .) # use '' for null string in $replace # ex: zsr 'type1' 'series A' use warnings; use strict; use File::Find; my ($search,$replace,$ext) = @ARGV; if (defined $ext) {$ext = ".$ext"} else {$ext = '.*'}; die "Usage : zsr 'search' 'replace' 'extension' (extension optional)\n +" if ($search eq ""); find (\&wanted, "."); sub wanted { my $open = $_; my $tempfile = 0; if (!($open =~ /$ext$/i) or (-d||-B||-l)) {return} print $open,"\n"; my $mode = (stat $open)[2]; #print $mode,"\n"; #printf "Permissions are %04o\n", $mode & 07777; open (TEMP,">> $tempfile"); open (FH, "< $open") or die "Can't open $open: $!\n"; while (<FH>) { $_ =~ s/$search/$replace/g; print TEMP $_; } close FH; close TEMP; rename ($tempfile, $open) or die "Can't rename $open: $!\n"; chmod ($mode,$open) or die "Can't restore permissions to $open, possib +ly wrong owner: $!\n"; }

    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: Searching and Replacing file content within directory
by roboticus (Chancellor) on Nov 12, 2010 at 16:10 UTC

    King0:

    I'd suggest you be sure to not update URLs that already have a port specification, or you'll wind up with junk like "http://localhost:8080:8080".

    ...roboticus

Re: Searching and Replacing file content within directory
by AR (Friar) on Nov 12, 2010 at 14:18 UTC
    Can you use sed? It can make replacements in-line and search recursively through the directories.
      I love perl yet often I'll see an issue like this and think 'replace that with a small shell script' or in this case do it all with sed. Glad I'm not the only one. :)

        sed? I prefer to stick with perl. Here's a 1-liner to search and replace:

        perl -i -wpe 's/INPUT_TXT/OUTPUT_TXT/g' file1 [... fileN]

        See the perlrun docs for info on using -i.

        Use find if needed to generate the list of files, and pipe its output to Perl via xargs.