kiat has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I've a couple of questions regarding the processing of a flat file using a perl cgi program. The program needs to edit, append and delete entries in the file. My questions are:

1) Does the fact that the database is a flat file make the processing inherently error prone?

2) If the flat file gets very large, does it increase the probability of an error during processing?

3) Does using a structured database (database files created using, say, MySql) make the processing safer?

I hope someone can enlighten me on the above. Thanks in anticipation :)

kiat

Replies are listed 'Best First'.
(Ovid) Re: perl cgi and flat textfile
by Ovid (Cardinal) on Dec 07, 2001 at 05:18 UTC

    I am tempted to simply say "yes" to all three of your questions, but that really wouldn't be fair. It really depends on your coding skill and how much extra work you want to put into it. Using one-liners, here are basic steps:

        # deleting entries
    perl -ni -e 'print if $_ !~ /old_stuff/' somefile.txt

    # editing entries
    perl -pi -e 's/new_stuff/old_stuff/' somefile.txt

    # adding entries :)
    cat somefile.txt new_records.txt > new_file.txt

    [download]

    As you can see, nothing in the above list is difficult, it's all in what you really need to do. However, incorporating such things into a program can sometimes be difficult. If you must use flat files, I would strongly recommend using DBD::CSV. Then, if you need to switch to a database, the conversion is trivial.

    If you do decide to use a CSV database (with or without DBD::CSV), you can use my CSV Database Validation program which does a fairly rigorous job of ensuring the integrity of your data. You can use it to define basic data types, foreign key constraints and even match data against regular expressions. Create a schema, create your data and put your application through its paces. Then, run the validation script against it to ensure the integrity of your data.

    I include full POD documentation in the program including a small tutorial.

    Cheers,
    Ovid

    Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

[reply]
[d/l]
      So what's the cause of those errors
[reply]

        You've replied to a 12 year old post. No errors are reported in the post you have responded to. If you're having issues perhaps it's best to start a new thread.

[reply]
Re: perl cgi and flat textfile
by sparkyichi (Deacon) on Dec 07, 2001 at 05:10 UTC
    I have found that error prone programming causes processing to be inherently error prone. My experience is that the internals of your programming (Code, Source Files, and Output) should have no errors or inconsistencies. Sometimes you might get an error from external sources such as running out of disk space or memory. Unless your program causes theses errors then there is nothing you can really do about that. It is your job as a programmer to insure that your code is as fool proof as you can make it.

    I have found that if I can, I like to cerate my own Perl Database out of a text file. I guess it comes from being lazy (hey, a Perl hacker is suppose to be lazy) because connecting to a database such as Oracle, MySQL, MSAccess cause more overhead for my scripts. You would need to anticipate more things that might go wrong such as connection time-out, table unavailability and the like. If it is just a text file somewhere, then you can control the file more completely through Perl (If you need to add a table, just create a new file or what ever.) By the same token though there are times where you would need to use a managed database such as many other users and applications that need to access the data in many different ways. If you are having problems with your coding you can look at some of the many DBI modules or look in the Camel and Panther books where there are examples and explanations on how to make a working database out of a file.

    Here is an example where I just write an Excel spreadsheet (Untested Sample code):
    use Spreadsheet::WriteExcel;
my $julian = 'Header';
my $ttime  = '1208011500';
my $workbook = Spreadsheet::WriteExcel->new("$julian.xls");
    $worksheet   = $workbook->addworksheet("Missing Stores $julian");
    $format      = $workbook->addformat();
    $worksheet->set_column(0, 0, 12); 
    $format->set_align('center');
    $format->set_bold(1);
    $worksheet->write(0, 0, "Julian/Time", $format);
    $worksheet->write(0, 1, "Store", $format);
    $format->set_bold(0);
    $worksheet->write(1, 0, "$julian/$ttime", $format);
    my $i = 1;
    my $n = 0;

    until ( $n == $missxn ){
        $worksheet->write($i, 1, "@No_file_ssc[$n]", $format);
        $i++;
        $n++;
    }

    [download]
     @No_file_ssc is declaired else where in the script.

    Sparky
[reply]
[d/l]
Re: perl cgi and flat textfile
by trantor (Chaplain) on Dec 07, 2001 at 16:29 UTC

    I agree with the previous posts, and I just wanted to point out something more.

    When dealing with CGIs and text file databases, locking is highly recommended, because of the nature of HTTP requests themselves.

    You should also consider users pressing submitting many times or reloading the page, they both impact negatively your flat file DB in case operations that change data are involved.

    I personally like to use text files when it makes sense, e.g. when data needs to be implicitly human readable or when dealing with pre-existing databases and file formats.

    In other situations I tend to prefer DBM files, woinderfully supported by perl (e.g. MLDBM) and interoperable as well, or the DBI interface, which can operate on text files when/if needed, as already pointed out.

    -- TMTOWTDI

[reply]
Re: perl cgi and flat textfile
by eformat (Initiate) on Dec 07, 2001 at 20:13 UTC
    Hi,
    just as an aside, i've had very good experiences using DBI and DBD::CSV (a database driver for CSV(or tab separated) files).
    Using this apporach allows you to code using familiar sql type statements, while interfacing to a flat file. If later on you want to swap to a db like MySQL no problem, simply swap the DBI driver you're using.
    Mike
[reply]

Back to Seekers of Perl Wisdom