in reply to Re^5: Database in a folder?
in thread Database in a folder?

So, uncached, it is a relatively trivial thing to implement, but will be very, very slow.

Thank you for your deep analysis. This solved my problem to find suitable tie module.

I understand now better why this is not so trivial (as I presumed) to get an universal solution. The problem with tied system means lots of unnecessary file access and caching means problems with multitasking.

I decided to go with my own solution. My program practically keeps all data in memory but when data it is updated it writes the whole record to the file after the record is fully updated.

open (DFH,">",DATADIR.$key) or die;
print DFH "$_=>$record{$_}\n" foreach (keys %record);
close DFH;

[download]

When the program loads it scans the directory and reads all records in memory. At this time I do not need sharing between processes so I am using shared variables for threads. Otherwise I should check flocks and timestamps every time when I access the record again.

open (DFH,"<",DATADIR.$key) or die $key;
my %record = split(/=>|\n/o,<DFH>);
close DFH;

[download]

This way the last available data is always in file system and is user modifiable between runs.

Replies are listed 'Best First'.
Re^7: Database in a folder?
by BrowserUk (Patriarch) on Feb 18, 2010 at 23:23 UTC
    when data it is updated it writes the whole record to the file

    That's a nice compromise for data integrity purposes.

    A possible enhancement, as your data is in shared memory, would be to start another background thread with a queue and offload the write-back to disk from your processing threads, by queueing the primary key (filename) of updated records.

    When the program loads it scans the directory and reads all records in memory. At this time I do not need sharing between processes

    As long as your data continues to fit in memory, and your only operating on it from one processor that seems like an effective strategy.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom