mmartin has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

As the title suggests, I wanted to see if there was anyway to issue only one open command, and be able to capture ALL the
file's contents, then overwrite the file... Is this possible?

I know using ">" or write will clobber/remove any contents in the file once you issue the open command with '>' as the
mode. And "<" or read opens the file in read mode. And, doing ">>" will open in append mode to write to the end of the
file... I also know you can add "+" to any of those symbols above to add the opposite mode on top of what you already
used. *i.e. ">+" write+read, "+<" read+write, "+>>" append+read, etc....

So I'm curious if there is a special way that I can read/capture the contents of a file first, then overwrite that same
file while ONLY open'ing the file once..? Is that possible?

In case your wondering what my goal is, this is what I'm trying to accomplish: Basically, the file in question will be a
log file. And I really only want the file to contain 21 lines (*line 1 is the column headings). So lets say the file has
21 lines currently in it. What I want to do is Remove the top line of data (*NOT the header line), so remove line 2. Then
add a new line of new data to the end of the file, sort of like rotating the file's contents.

I know I can accomplish this by opening the file in read mode, slurping it's contents into an array then adding the new
line of data to that array, close the file (*the Array would contain 22 elements at this point). Then, re-open the file in
write mode (*clobber contents), loop through the array and print each element back to that same file while skipping
element [1]...

Any thoughts or suggestions would be much appreciated!

Thanks in Advance,
Matt

  • Comment on Capture Contents AND Overwrite without Opening Twice?

Replies are listed 'Best First'.
Re: Capture Contents AND Overwrite without Opening Twice?
by roboticus (Chancellor) on Oct 08, 2014 at 19:38 UTC

    [mmartin:

    It's pretty simple, as the AMs have indicated. I wrote a quickie program to illustrate:

    $ perl tuv.pl $ cat tuv.pl print $FH reverse @lines; seek $FH, 0, 0; my @lines = <$FH>; open my $FH, '+<', $0; use warnings; use strict;

    I could've printed the program *before* running it, but (a) I'd've had to print it twice to show that it did anything, and (b) it would have been less amusing. ;^)

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      I love it :-) It gets even better if you change $FH to a bareword filehandle FH, then the resulting code remains runnable, albeit useless ;-)

Re: Capture Contents AND Overwrite without Opening Twice?
by MidLifeXis (Monsignor) on Oct 08, 2014 at 20:01 UTC

    Also watch for race conditions between the open and the truncate/write. You could end up with a mangled log file or a lost update.

    --MidLifeXis

Re: Capture Contents AND Overwrite without Opening Twice?
by dasgar (Priest) on Oct 08, 2014 at 21:13 UTC

    Not sure if it will do exactly what you want, but I'm thinking that Tie::File might work. You open the file for read/write access when you tie the file to an array. Then using array manipulations, you can add/remove/modify the contents of the file. Then the file is closed when you untie the file.

      To wit: test.pl

      use strict; use warnings; use Tie::File; my $infile = shift; tie my @lines, 'Tie::File', $infile; printf "File has %d lines\n",$#lines+1; print "$_\n" for @lines; # clobber file @lines = (); printf "\nFile has %d lines\n",$#lines+1; print "$_\n" for @lines; # insert replacement lines push @lines, "replacement line here"; push @lines, "another new line"; printf "\nFile has %d lines\n",$#lines+1; print "$_\n" for @lines; untie @lines;

      Let's run it.

      $ perl test.pl test.file File has 4 lines one two three four File has 0 lines File has 2 lines replacement line here another new line

      Verify that we have in fact changed the file:

      $ cat test.file replacement line here another new line $

      Updated for readability

      1 Peter 4:10
Re: Capture Contents AND Overwrite without Opening Twice?
by Anonymous Monk on Oct 08, 2014 at 19:23 UTC

      Make sure to read the docs: Check the return values of the functions (... or die "truncate failed"; etc.), and note that truncate has some kinks in Windows.

Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 08, 2014 at 22:01 UTC
    Hey All, thanks for the new replies...

    Sorry, but I am soo confused right now... Maybe it's because I'm at the end of my day, but I am just not grasping how this is supposed
    to work? The one example that was given by roboticus, to me, almost seems like its written in reverse. I'm sure it's not because I know
    you guys are the experts, but I guess I'm just totally lost here...

    From what I read (*at perldoc.perl.org), it sounds like you can use 'seek' to move around the File, but in bytes or characters and not
    lines (right?). Something like, you pass it a number as the second argument to seek, positive num to move forward x bytes or
    negative to move in reverse -x bytes.

    And then truncate can be used to truncate or remove portions of a file to a specific length. I also read about 'tell' which will tell you
    where in the file you are currently reading from, correct (*not sure if I need that or not..)?


    If possible, could someone explain (*in psudeo-code) how exactly it SHOULD work...? Something like:
    (*use open() here) - Open the filehandle FH (*use array and FH) - Slurp all lines from FH to Array (*use seek here) - Go to a position in the file (*etc....)
    I'm just not grasping the logic behind it I guess...

    Or maybe an example. Let's say I have the File "test_file.log" containing:
    COL-1 COL-2 COL-3 COL-4 Line #1 Line #2 Line #3 Line #4 Line #5
    And basically, that log file should never have more then 6 TOTAL lines (*5 lines containing the data, and +1 for the Col headings), and the
    newest data would be at the bottom.

    And if I ran my script and added a NEW line of data to the file, the end result should be:
    COL-1 COL-2 COL-3 COL-4 Line #2 Line #3 Line #4 Line #5 Line #6
    I don't know if that example changes any of your understanding's of my original post, sorry if I'm being a pain... I'm just trying to grasp how
    exactly seek and truncate can be used together on this...? I'm going to be heading home in a minute or 2 so maybe I'll understand better tomorrow when my head is a bit fresher...

    Again, sorry for the confusion on my part, it's been a long day..!
    And thanks AGAIN EVERYBODY for the replies, it is VERY much appreciated!

    Thanks,
    Matt

      roboticus's example is a script that reverses itself (it opens $0), and it was run first, then printed, so it came out in reverse :-)

      seek works in bytes, not characters or lines. You understood correctly that you can use it to move the current position around in the file (relative to its beginning, the current position, or its end). truncate cuts down the file to a certain size (AFAIK also in bytes, definitely not lines), but always from the beginning of the file.

      I understand you want to preserve the first line of the file. You could do that by figuring out where that line ends (in bytes!), and truncate the file to there. However, since we're only talking about one line here, the logic would be much easier if you just clobber the entire file, modify the array of lines, and write everything back out.

      my $MAXLINES=20; # not including header open my $fh, '+<', 'foo.txt' or die $!; my @lines = <$fh>; splice @lines, 1, @lines-$MAXLINES if @lines>$MAXLINES; push @lines, "newline\n"; truncate $fh, 0 or die "truncate failed"; seek $fh, 0, 0 or die "seek failed"; print $fh @lines; close $fh;

      But I would also second dasgar's suggestion for Tie::File.

      use Tie::File; my $MAXLINES=20; # not including header tie my @lines, 'Tie::File', 'foo.txt' or die "tie failed"; splice @lines, 1, @lines-$MAXLINES if @lines>$MAXLINES; push @lines, "newline\n"; untie @lines;
      Well you could use the -i flag on the command line to modify a file, well at least in a one-liner.

      Otherwise, I would not recommend opening files in read-and-write mode. But if you do want to do it this way, then you need to read the full file first and save the content in an array, make whatever changes you need to the array, and then only write back the array to the file. It works, but that would not be my recommended course of actions.

      I would suggest reading the input file (read mode), writing to a copy of it (write mode), and then, if everything went OK, to do the house cleaning, i.e. deleting the old file (or archiving it under another name), and renaming the new one to what you need.

        This is an important consideration iff the file will be accessed by other processes during the run of the script, or if it's important that the file doesn't get corrupted by either the script dieing or getting shot down externally while it is modifying the file. (probably what MidLifeXis's comment was aimed at as well)

Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 08, 2014 at 20:01 UTC
    Hey Anonymous Monk and Roboticus, thanks so much for the quick replies!

    Awesome, I've never used/seen truncate and seek commands in Perl before, so thank you for pointing
    me in the right direction...!

    Also, thanks for the example. That was much more then I expected so thanks for taking the time to do
    that, many thanks!

    Ok, I'll give it a shot after I check out truncate and seek in perldocs. And I'll send a reply back
    once I've gotten my code working...

    Thanks again for the examples and suggestions, very much appreciated!

    Thanks Again,
    Matt
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 17:49 UTC
    Hey Guys,

    So I got the Tie::File working in my script, in the previous post I had it working in a test script.

    Below is the section of my script where I use the tie() function. Basically, my script opens a socket and sits
    and waits for a remote server to make a connection. Once the socket connection is made, my script captures
    the data sent from the remote server (*Hostname, IP Addr, Timestamp, and Location Name). Before the code
    below I check all the contents of those 4 variables. In the code below, I format a New Line of data (*$new_data),
    then I check and see if either the File/Array is empty (== -1) or if the 1st element in the Array does not contain
    the Header Text. If it doesn't, I use unshift and add it to the start of the array/file.

    I have run a few tests and it all seems to be working very well... I'm really liking the Tie::File Module, so thanks
    for showing that to me.!

    ### Check and make sure ALL the variables contain values and are NOT = += ERROR: if ($ipAddr !~ /ERROR/i && $hostname !~ /ERROR/i && $location !~ /ERRO +R/i && $timestamp !~ /ERROR/i) { ### Declare the Log File: my $LOG_FILE = "$LOG_DIR/$hostname.log"; ### build the line to print to the LogFile: # *MAX COLUMN WIDTHS --> Timestamp(24) Location(14) IP Addr +ess(15) Hostname(18) my $new_data = sprintf "%-25s %-15s %-16s %-19s", "$timestam +p", "$location", "$ipAddr", "$hostname"; ### Open the Log File and "TIE" it to the array @logData: # *With this (*Tie::File) whatever happens to the array is refl +ected in the file: tie my @logData, 'Tie::File', "$LOG_FILE" or die "Error: tie faile +d, $!"; ### If the File/Array is empty --OR-- THe first element does not c +ontain the LOG_HEADER, then... if ($#logData == -1 || $logData[0] !~ /LAST CHECK|LOCATION|IP ADDR +ESS|HOSTNAME/gi) { ### Insert the LOG_HEADER as element '0' in the array: unshift @logData, "$LOG_HEADER"; } ### If @logData has more lines then MAXLINES, then Splice out X li +nes starting at Line 2 (*i.e. index '1') splice @logData, 1, @logData-$MAXLINES if @logData>$MAXLINES; ### Next, push the new data onto the end of the Array/File: push @logData, "$new_data\n"; ### Untie the Array from the Log File (*essentially closing the fi +le...): untie @logData; }

    So, thanks AGAIN everybody for the help, very much appreciated!!

    Thanks,
    Matt

      Looks good, thanks for sharing!

      (The only very minor nit I might pick is that you don't need to interpolate your variables into a string in a couple of places, such as "$LOG_FILE"; just saying $LOG_FILE is fine.)

Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 14:31 UTC
    Hey Everybody, thanks for the replies... Much Appreciated!

    For the questions about any other processes accessing this file... No, no other processes will touch the file. The file
    will only be used by my script.

    Ok gotcha, I see what you mean about roboticus' example. That makes sense then...

    I guess I will do the way I mentioned in my OP. About saving all the file's contents into an array and then modifying the
    array and printing that back to the file. I definitely understand how to do it that way.

    Also, thanks for the example with the Tie::File module. I have not used that one before, so I'll check out the docs
    for that and see how that works.

    Thanks again everybody for the replies, very much appreciated!

    Thanks Again,
    Matt
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 15:25 UTC
    Hey All,

    So, I just gave the Tie::File module a try using AM's example, and it worked perfectly!

    If I understand it correctly, "untie" will untie the array to the file (*unlocking the file), in
    essence would that be the same as "closing" the file?

    But anyway, thanks again everybody for the help.

    Thanks,
    Matt
      If I understand it correctly, "untie" will untie the array to the file (*unlocking the file), in essence would that be the same as "closing" the file?

      untieing a Tie::File array will close the file. However, Tie::File will not do any file locking for you unless you ask it to via its ->flock method (also make sure to read up on flock about its details and caveats, such as only being advisory locking). Your OS or FS might do some locking of its own, however on a normal *NIX system that's unlikely.

      dasgar suggested Tie::File.

      1 Peter 4:10