Re: Capture Contents AND Overwrite without Opening Twice?
by roboticus (Chancellor) on Oct 08, 2014 at 19:38 UTC
|
[mmartin:
It's pretty simple, as the AMs have indicated. I wrote a quickie program to illustrate:
$ perl tuv.pl
$ cat tuv.pl
print $FH reverse @lines;
seek $FH, 0, 0;
my @lines = <$FH>;
open my $FH, '+<', $0;
use warnings;
use strict;
I could've printed the program *before* running it, but (a) I'd've had to print it twice to show that it did anything, and (b) it would have been less amusing. ;^)
...roboticus
When your only tool is a hammer, all problems look like your thumb. | [reply] [d/l] |
|
|
| [reply] [d/l] [select] |
Re: Capture Contents AND Overwrite without Opening Twice?
by MidLifeXis (Monsignor) on Oct 08, 2014 at 20:01 UTC
|
Also watch for race conditions between the open and the truncate/write. You could end up with a mangled log file or a lost update.
| [reply] |
Re: Capture Contents AND Overwrite without Opening Twice?
by dasgar (Priest) on Oct 08, 2014 at 21:13 UTC
|
Not sure if it will do exactly what you want, but I'm thinking that Tie::File might work. You open the file for read/write access when you tie the file to an array. Then using array manipulations, you can add/remove/modify the contents of the file. Then the file is closed when you untie the file.
| [reply] |
|
|
use strict;
use warnings;
use Tie::File;
my $infile = shift;
tie my @lines, 'Tie::File', $infile;
printf "File has %d lines\n",$#lines+1;
print "$_\n" for @lines;
# clobber file
@lines = ();
printf "\nFile has %d lines\n",$#lines+1;
print "$_\n" for @lines;
# insert replacement lines
push @lines, "replacement line here";
push @lines, "another new line";
printf "\nFile has %d lines\n",$#lines+1;
print "$_\n" for @lines;
untie @lines;
Let's run it.
$ perl test.pl test.file
File has 4 lines
one
two
three
four
File has 0 lines
File has 2 lines
replacement line here
another new line
Verify that we have in fact changed the file:
$ cat test.file
replacement line here
another new line
$
Updated for readability
| [reply] [d/l] [select] |
Re: Capture Contents AND Overwrite without Opening Twice?
by Anonymous Monk on Oct 08, 2014 at 19:23 UTC
|
| [reply] [d/l] [select] |
|
|
| [reply] [d/l] |
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 08, 2014 at 22:01 UTC
|
Hey All, thanks for the new replies...
Sorry, but I am soo confused right now... Maybe it's because I'm at the end of my day, but I am just not grasping how this is supposed
to work? The one example that was given by roboticus, to me, almost seems like its written in reverse. I'm sure it's not because I know
you guys are the experts, but I guess I'm just totally lost here...
From what I read (*at perldoc.perl.org), it sounds like you can use 'seek' to move around the File, but in bytes or characters and not
lines (right?). Something like, you pass it a number as the second argument to seek, positive num to move forward x bytes or
negative to move in reverse -x bytes.
And then truncate can be used to truncate or remove portions of a file to a specific length. I also read about 'tell' which will tell you
where in the file you are currently reading from, correct (*not sure if I need that or not..)?
If possible, could someone explain (*in psudeo-code) how exactly it SHOULD work...? Something like:
(*use open() here) - Open the filehandle FH
(*use array and FH) - Slurp all lines from FH to Array
(*use seek here) - Go to a position in the file
(*etc....)
I'm just not grasping the logic behind it I guess...
Or maybe an example. Let's say I have the File "test_file.log" containing:
COL-1 COL-2 COL-3 COL-4
Line #1
Line #2
Line #3
Line #4
Line #5
And basically, that log file should never have more then 6 TOTAL lines (*5 lines containing the data, and +1 for the Col headings), and the
newest data would be at the bottom.
And if I ran my script and added a NEW line of data to the file, the end result should be:
COL-1 COL-2 COL-3 COL-4
Line #2
Line #3
Line #4
Line #5
Line #6
I don't know if that example changes any of your understanding's of my original post, sorry if I'm being a pain... I'm just trying to grasp how
exactly seek and truncate can be used together on this...? I'm going to be heading home in a minute or 2 so maybe I'll understand better tomorrow when my head is a bit fresher...
Again, sorry for the confusion on my part, it's been a long day..!
And thanks AGAIN EVERYBODY for the replies, it is VERY much appreciated!
Thanks,
Matt
| [reply] [d/l] [select] |
|
|
roboticus's example is a script that reverses itself (it opens $0), and it was run first, then printed, so it came out in reverse :-)
seek works in bytes, not characters or lines. You understood correctly that you can use it to move the current position around in the file (relative to its beginning, the current position, or its end). truncate cuts down the file to a certain size (AFAIK also in bytes, definitely not lines), but always from the beginning of the file.
I understand you want to preserve the first line of the file. You could do that by figuring out where that line ends (in bytes!), and truncate the file to there. However, since we're only talking about one line here, the logic would be much easier if you just clobber the entire file, modify the array of lines, and write everything back out.
my $MAXLINES=20; # not including header
open my $fh, '+<', 'foo.txt' or die $!;
my @lines = <$fh>;
splice @lines, 1, @lines-$MAXLINES if @lines>$MAXLINES;
push @lines, "newline\n";
truncate $fh, 0 or die "truncate failed";
seek $fh, 0, 0 or die "seek failed";
print $fh @lines;
close $fh;
But I would also second dasgar's suggestion for Tie::File.
use Tie::File;
my $MAXLINES=20; # not including header
tie my @lines, 'Tie::File', 'foo.txt' or die "tie failed";
splice @lines, 1, @lines-$MAXLINES if @lines>$MAXLINES;
push @lines, "newline\n";
untie @lines;
| [reply] [d/l] [select] |
|
|
Well you could use the -i flag on the command line to modify a file, well at least in a one-liner.
Otherwise, I would not recommend opening files in read-and-write mode. But if you do want to do it this way, then you need to read the full file first and save the content in an array, make whatever changes you need to the array, and then only write back the array to the file. It works, but that would not be my recommended course of actions.
I would suggest reading the input file (read mode), writing to a copy of it (write mode), and then, if everything went OK, to do the house cleaning, i.e. deleting the old file (or archiving it under another name), and renaming the new one to what you need.
| [reply] |
|
|
This is an important consideration iff the file will be accessed by other processes during the run of the script, or if it's important that the file doesn't get corrupted by either the script dieing or getting shot down externally while it is modifying the file. (probably what MidLifeXis's comment was aimed at as well)
| [reply] [d/l] |
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 08, 2014 at 20:01 UTC
|
Hey Anonymous Monk and Roboticus, thanks so much for the quick replies!
Awesome, I've never used/seen truncate and seek commands in Perl before, so thank you for pointing
me in the right direction...!
Also, thanks for the example. That was much more then I expected so thanks for taking the time to do
that, many thanks!
Ok, I'll give it a shot after I check out truncate and seek in perldocs. And I'll send a reply back
once I've gotten my code working...
Thanks again for the examples and suggestions, very much appreciated!
Thanks Again,
Matt
| [reply] |
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 17:49 UTC
|
Hey Guys,
So I got the Tie::File working in my script, in the previous post I had it working in a test script.
Below is the section of my script where I use the tie() function. Basically, my script opens a socket and sits
and waits for a remote server to make a connection. Once the socket connection is made, my script captures
the data sent from the remote server (*Hostname, IP Addr, Timestamp, and Location Name). Before the code
below I check all the contents of those 4 variables. In the code below, I format a New Line of data (*$new_data),
then I check and see if either the File/Array is empty (== -1) or if the 1st element in the Array does not contain
the Header Text. If it doesn't, I use unshift and add it to the start of the array/file.
I have run a few tests and it all seems to be working very well... I'm really liking the Tie::File Module, so thanks
for showing that to me.!
### Check and make sure ALL the variables contain values and are NOT =
+= ERROR:
if ($ipAddr !~ /ERROR/i && $hostname !~ /ERROR/i && $location !~ /ERRO
+R/i && $timestamp !~ /ERROR/i)
{
### Declare the Log File:
my $LOG_FILE = "$LOG_DIR/$hostname.log";
### build the line to print to the LogFile:
# *MAX COLUMN WIDTHS --> Timestamp(24) Location(14) IP Addr
+ess(15) Hostname(18)
my $new_data = sprintf "%-25s %-15s %-16s %-19s", "$timestam
+p", "$location", "$ipAddr", "$hostname";
### Open the Log File and "TIE" it to the array @logData:
# *With this (*Tie::File) whatever happens to the array is refl
+ected in the file:
tie my @logData, 'Tie::File', "$LOG_FILE" or die "Error: tie faile
+d, $!";
### If the File/Array is empty --OR-- THe first element does not c
+ontain the LOG_HEADER, then...
if ($#logData == -1 || $logData[0] !~ /LAST CHECK|LOCATION|IP ADDR
+ESS|HOSTNAME/gi)
{
### Insert the LOG_HEADER as element '0' in the array:
unshift @logData, "$LOG_HEADER";
}
### If @logData has more lines then MAXLINES, then Splice out X li
+nes starting at Line 2 (*i.e. index '1')
splice @logData, 1, @logData-$MAXLINES if @logData>$MAXLINES;
### Next, push the new data onto the end of the Array/File:
push @logData, "$new_data\n";
### Untie the Array from the Log File (*essentially closing the fi
+le...):
untie @logData;
}
So, thanks AGAIN everybody for the help, very much appreciated!!
Thanks,
Matt
| [reply] [d/l] |
|
|
Looks good, thanks for sharing!
(The only very minor nit I might pick is that you don't need to interpolate your variables into a string in a couple of places, such as "$LOG_FILE"; just saying $LOG_FILE is fine.)
| [reply] [d/l] [select] |
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 14:31 UTC
|
Hey Everybody, thanks for the replies... Much Appreciated!
For the questions about any other processes accessing this file... No, no other processes will touch the file. The file
will only be used by my script.
Ok gotcha, I see what you mean about roboticus' example. That makes sense then...
I guess I will do the way I mentioned in my OP. About saving all the file's contents into an array and then modifying the
array and printing that back to the file. I definitely understand how to do it that way.
Also, thanks for the example with the Tie::File module. I have not used that one before, so I'll check out the docs
for that and see how that works.
Thanks again everybody for the replies, very much appreciated!
Thanks Again,
Matt
| [reply] |
Re: Capture Contents AND Overwrite without Opening Twice?
by mmartin (Monk) on Oct 09, 2014 at 15:25 UTC
|
Hey All,
So, I just gave the Tie::File module a try using AM's example, and it worked perfectly!
If I understand it correctly, "untie" will untie the array to the file (*unlocking the file), in
essence would that be the same as "closing" the file?
But anyway, thanks again everybody for the help.
Thanks,
Matt
| [reply] |
|
|
If I understand it correctly, "untie" will untie the array to the file (*unlocking the file), in
essence would that be the same as "closing" the file?
untieing a Tie::File array will close the file. However, Tie::File will not do any file locking for you unless you ask it to via its ->flock method (also make sure to read up on flock about its details and caveats, such as only being advisory locking). Your OS or FS might do some locking of its own, however on a normal *NIX system that's unlikely.
| [reply] [d/l] [select] |
|
|
| [reply] |
|
|
| [reply] |
|
|