in reply to delete/over-write record in FlatFile

You may not think you have the budget to do it right, but do you have the budget to do it wrong? I honestly don't think this is going to be easier to do wrong!

But back to the question. To "edit" a file like this you basically have to load the entire thing into memory, make your changes, and then write the whole thing back out. And you'll have to lock the file while you're doing it to make sure you don't get two editors working at the same time. The reason it's not possible to edit the file in-place is because the lines are of variable length, so editing means changing the length of the file and the position of lines lower down.

So, you'll start by loading all the data into a structure in memory. Something like (untested):

open (DB,"ext.log") or die("There was a problem opening the file."); my @data; while (my $entry=<DB>) { my @fields = split(/ /,$entry); push @data, \@fields; } close DB;

Now you'll make a change:

  $data[5][1] = "New value!";

Then write it all back out again:

open(DB, ">", "ext.log") or die $!; foreach my $row (@data) { my $line = join(' ', @$row); print DB $line, "\n"; } close DB;

If your file is it at all big this is going to totally suck. In that case you might look at using something like Tie::File. That will help with memory usage, but there's no helping the fact that making random edits to a sequential file will be slow.

Now doesn't this make you want to do it right? Good! You should be using a database of some kind to hold the data. If you have a real DB like MySQL or Postgres already available then use that. If not you might look at something like DBD::SQLite to save you the trouble of getting a real DB installed. Then create a table, load your log data into it, and suddenly editing lines will be cake! And who doesn't like cake?

-sam

Replies are listed 'Best First'.
Re^2: delete/over-write record in FlatFile
by lakeTrout (Scribe) on Mar 25, 2008 at 21:43 UTC
    Hi Sam,
    That's what I meant by right -- having a "real" DB and the use to tools/mods but I'm stuck with what I have. As for locking people out, I was going to use a C-style flock() but not sure yet. I know this will totally suck, that I can agree with! I will say after the dataset is complete there will be ZERO chance to simultaneous edits to the DB, so locking isn't a huge concern at this point. I'll try to make what you have there work, but like I said -- I'm not really much of a developer, I just try to help when I can. Thank you sir!

    lakeTrout

      You could write two programs. One takes what you have in the flatfile and inserts it into the database. The second does what you need to do out of the database.

      Before you object, consider that it's effectively the same amount of work as what you're doing now. The difference is that it'll be much easier to work with data when it's in a database. (You'll have to figure out how to map back and forth between Perl and your flatfile, which is what the flatfile-to-database converter has to do anyway.)

      "Stuck" huh? That sounds unpleasant. Should we call the police? Are you being held against your will?

      I guess I'm just an asshole, but I can't imagine anyone successfully preventing me from using a technique I knew was superior and which wouldn't take any additional time to implement! What's the worst they can do? Fire you? It sounds like they'd be doing you a favor!

      -sam