Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

writing to the top of a file

by Anonymous Monk
on May 20, 2004 at 02:43 UTC ( [id://354830]=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I know how to open a file for appending, but how do you write to the TOP of it? I have a CGI form tha writes to a text file and I would like the newer information on top rather than on bottom so we don't have to scroll to the bottom to find out what's new.

What's the easiest way to do this?

Replies are listed 'Best First'.
Re: writing to the top of a file
by broquaint (Abbot) on May 20, 2004 at 02:57 UTC
    Due to the nature of the structure of files you can't write to the top of them. So a simple solution is to create a new file and overwrite the old file e.g
    open( my $old, '<', 'current.log' ) or die "ack: $!"; open( my $new, '>', 'current.log.new' ) or die "ack: $!"; print {$newfh} $your_data_here; print {$newfh} $_ while <$old>; close $old; close $new; rename 'current.log.new', 'current.log';
    HTH

    _________
    broquaint

Re: writing to the top of a file
by coec (Chaplain) on May 20, 2004 at 03:07 UTC
    Try this
    #!/usr/bin/perl use strict; use warnings; use Tie::File; use Data::Dumper; my $file = "file.txt"; my @lines; #tie @lines, 'Tie::File', $file || die $!; tie @lines, 'Tie::File', $file or die $!; my @tmp = splice @lines; push @lines, <STDIN>; push @lines, @tmp; untie @lines;
    Tie::File isn't a standard module but its easily installed as it is pure Perl (i.e. doesn't need a C compiler installed).

    CC

    Updated: changed '||' to 'or'.

    Another update

    Given that whatever solution you choose will be running from a web page (via CGI), have you given any thought file locking? What if multiple copies of the CGI script are running at the same time? You could potentially lose a lot of data...

      unshift makes it simpler:
      #!/usr/bin/perl use strict; use warnings; use Tie::File; my $file = "file.txt"; my $insert = shift @ARGV; chomp $insert; my @lines; tie @lines, 'Tie::File', $file or die $!; unshift @lines, $insert; # more simple, isn't it? untie @lines;
      best regards,
      neniro

      Tie::File is core as of perl 5.8.

      Using Tie::File this way will still need to read and write the whole file. This is much slower than simply reading to the end to get the last line.

      It also doesn't scale well for largish files, and doesn't address the concurrency issue.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
Re: writing to the top of a file
by BrowserUk (Patriarch) on May 20, 2004 at 03:36 UTC

    Unless reads from this file are considerably more frequent than writes, reading the whole file and then re-writing it with the new stuff at the top in order to speed up reads will have dubious benefit.

    If the reads and writes are paired (equal numbers of each) then one cycle of 'Writing to the end of file + reading the whole file to get the last line' compares favourably with 'Reading the whole file, writing the new line and then re-writing the whole file + reading the first line'.

    You would almost certainly be better of writing new stuff to the end, and then using File::ReadBackwards or a similar technique to retrieve it.


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
      I concur. Even sucking the whole file in with @lines = <$INFILE>; and just printing the results in reverse order makes more sense than physically writing to the file in such an unnatural order. Of course, that presumes the file is not of a monstrous size, etc etc.
Re: writing to the top of a file
by saintmike (Vicar) on May 20, 2004 at 02:54 UTC
    By asking to write to the top of a file you're implying that you want to insert bytes at the top while the rest of the file moves down.

    Unrelated to Perl, filesystems typically don't provide functionality for that to happen transparently. What you need to do is read out the old file content into memory, insert the new data at the beginning (Perl's scalars or arrays are offering this in a nice transparent way) and then write the whole blob back to disk.

Re: writing to the top of a file
by graff (Chancellor) on May 20, 2004 at 03:37 UTC
    I think broquaint's solution will prove to be the most efficient, and will tend to impose the smallest load on resources. Of course, if the file gets really big, it'll take longer to re-write the whole thing each time you add a line or few at the top -- but the growth in delay will be relatively minor and nothing else will blow up, because you're not trying to hold the whole thing in memory (which is what would happen with coec's suggestion for using Tie::File. (There might be a way to use Tie::File that wouldn't involve holding the entire file contents in memory, but broquaint's approach is just easier.)

    Apart from that, you might consider looking at the problem a different way: keep the file i/o simple (always append new text at the end of the file, the way God intended), and just change how you read and manipulate the file data for display.

    If the people reading the display are only interested in the most recent content, you might decide that they won't want/need to look any farther than the "N" most recent lines. If you have the "tail" utility (everybody should have this by now), why not use it, since it was created to do just what you want (mostly):

    my @latest = reverse( `tail -$n $datafile` ); # @latest has the last $n lines from $datafile, last line first...
    If you're offended by the use of backticks, you might consider estimating how many bytes would likely cover $n lines, seek to that many bytes from the end of the file, and then read to the end of the file in the usual way assigning lines to array elements, and reverse the array; the oldest line might be just a fragment, but you could just ignore that one. (And $n could even be user-specified.)

    (update: As usual, BrowserUK has provided a more sensible and effective approach. I'd follow his advice.)

      Being retarded I'll ask a silly question or two:

      1) as coec said what about multiple instances of the script running. How best stop data loss? Is Perl's advisory locking flock sufficient?

      2) if the file gets really large, broquaints solution could (potentially) fill a file ssytem resouce.

      I am very new to Perl Monks and Perl and most other things IT. Just curious about these issues.

      retard

        1. Every time this issue comes up, I tend to favor using a semaphore file. Provided that we're talking about a file that is only updated by perl processes, and that all these processes follow the same procedure in terms of "asking" for access to update the file, then there's no problem. (Find an example of a semaphore file module here, which includes a reference to a very good article on locking any shared resource.)

        2. If the file size happens to be equal to or greater than the available free space on a drive, any solution for trying to "prepend" new data at the top of a file will overfill the file system, because you need to write the new file before you can delete the old one. That's a very good reason to avoid trying to update a file this way. Appending to a file will only fail if the amount of new data being added exceeds the amount of available free space on the drive.

        You're wise to be curious about these issues.

Re: writing to the top of a file
by gsiems (Deacon) on May 20, 2004 at 03:10 UTC

    How about something like:

    open (IN, $file) or die "$!\n"; my @ary = <IN>; close IN; open (OUT, ">$file") or die "$!\n"; print OUT "stuff\n"; print OUT @ary; close OUT;

    or

    rename $file, $tempfile; open (OUT, ">$file") or die "$!\n"; print OUT "stuff\n"; open (IN, $tempfile) or die "$!\n"; while (<IN>) { print OUT $_; } close IN; close OUT;

    Update: or better yet, see broquaint's post above.

Re: writing to the top of a file
by pg (Canon) on May 20, 2004 at 03:49 UTC

    Personally, I would go database, then the path to access data no longer depends on the way the data is stored, or be added, at least this is no longer your concern.

    I prefer to do the right thing, and not to spend the extra effort that could be easily avoided.

Re: writing to the top of a file
by Dr. Mu (Hermit) on May 20, 2004 at 05:58 UTC
    Make friends with seek and tell, and use a linked list. Here's how:
    1. Your file will be composed of variable-length records.
    2. Each record will be appended to the end of the file, as nature intended.
    3. The first n bytes of each record will contain the packed address of the previously-written record.
    4. The first n bytes of the first record (i.e. the first n bytes of the file) will contain the address of the last-written record.
    When writing a record,
    1. Read the address of the last record from the beginning of the file. Call it $last.
    2. seek to the end of the file.
    3. Use tell to see where you are. Call this address $next.
    4. Write $last to the end of the file, then your data record.
    5. seek to the beginning of the file, and write $next there.
    When reading the file,
    1. Read $last from the beginning of the file.
    2. seek to the end of the file, and use tell to get the current address. Call it $next.
    3. seek to $last, and read n bytes into the variable $prev.
    4. Read $next - $last - n bytes from the file. This is the last unread record.
    5. Set $last = $prev and $next = $last.
    6. Repeat from step 3. until you've read as many records as you want, or until $last > $next (i.e. you've reached the beginning of the file).
      You've turned a conceptually simple process into fifteen overly-complicated steps.

      When you think to use linked lists in Perl, you've probably overlooked a better way to do it using Perl's builtin features. Perl is so good at lists and hashes that you don't need a linked list - Perl's arrays can hold variable length elements, and can be manipulated much like linked lists.

      And packed string headers are most definately only a good idea in C. Regards your variable length 'records', you'd be better off keeping an index in a separate file. Much less complexity, and you can just append to the end of both files.

      But CPAN's Tie:: series of modules have probably already implemented the best solutions.

      ____________________
      Jeremy
      I didn't believe in evil until I dated it.

Re: writing to the top of a file
by jepri (Parson) on May 20, 2004 at 07:03 UTC
    Or you could save each submission to disk in a separate file, then build your webpage from all the files in the directory, starting with the most recently saved file.

    You don't have to put everything in the same file.

    ____________________
    Jeremy
    I didn't believe in evil until I dated it.

      Yes, but if each record is signficantly smaller than your file system's block size, you end up wasting a lot of space.
      A reply falls below the community's threshold of quality. You may see it by logging in.
Re: writing to the top of a file
by El Linko (Beadle) on May 20, 2004 at 09:48 UTC
    Since you only care about the record order when your viewing the records then you should probably do one of the following

    1: If you are a fan of the command line view the file using tail. This will show you the last X lines of the file.

    2. Write to a different file for each day/week/month(which one depends on expected website usage).

    3. Have a cron/somehow scheduled perl script to create a new file with the records ordered the way you want them and view this file.

    All of these have disadvantages but they allow you to keep the complexity out of the cgi script and are (probably/possibly) more efficient and less error prone.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://354830]
Approved by Steve_p
Front-paged by pbeckingham
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (3)
As of 2024-03-28 18:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found