in reply to Re^2: Storable - File empties itself?
in thread Storable - File empties itself?

If more than one process at a time may decide to rewrite the file you use for Storable data, you either need locks, or you need to delegate the problem (e.g. to a relational database).

For locking, see Re: Update config file parameters and flock. Note that locking can be messy if NFS is involved.

Often it is easier not to re-invent the wheel. Just don't use files at all, put your data into a relational database. File locking, concurrent reading and writing are solved problems for relational databases. You just need to connect to the database. That's what DBI is for.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Replies are listed 'Best First'.
Re^4: Storable - File empties itself?
by feumw (Sexton) on Apr 14, 2023 at 07:14 UTC
    I just posted this: Re: Storable - File empties itself?

    Yeah you might be right. I start checking if I can confirm my thoughts. Reworking things for now is an issue based on time. I'm almost on my own in 2 projects right now and yeah ... I go small steps and check if its two processes using 1 file first and see what I'm gonna do after.

      A 'minimal' code change solution would be to use DBI and DBD::SQLite (both install fine on Solaris btw), since you seem keen to avoid storing this in the database your CMS uses.

        A 'minimal' code change solution would be to use DBI and DBD::SQLite

        Yes, but -- Quoting https://www.sqlite.org/faq.html#q5:

        Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.

        SQLite uses reader/writer locks to control access to the database. [...] But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. [...]

        [...] SQLite allows multiple processes to have the database file open at once, and for multiple processes to read the database at once. When any process wants to write, it must lock the entire database file for the duration of its update. [...] that normally [...] takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. [...]

        However, client/server database engines (such as PostgreSQL, MySQL, or Oracle) usually support a higher level of concurrency and allow multiple processes to be writing to the same database at the same time. This is possible in a client/server database because there is always a single well-controlled server process available to coordinate access. If your application has a need for a lot of concurrency, then you should consider using a client/server database. [...]

        [...]

        So yes, SQLite supports multiple processes opening the same database. But unlike a "real" database server, it simply locks the entire database while writing. If writes are rare and quick, that normally should not be a problem. If writes are slow and/or happen often, readers may get into trouble with SQLite.

        you seem keen to avoid storing this in the database your CMS uses.

        This is a very good point. It should be easy to create a separate user and a separate table space in the existing database server for the perl scripts. Essentially, log in as admin, run one command to create a user and a second command to create a tablespace for that user. Perhaps you need to run a third command to grant privileges. After that, just use DBI to access that new tablespace.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        >> since you seem keen to avoid storing this in the database your CMS uses. Yes that's kinda true since our CMS doesn't use a database :-D We're about to upgrade and migrate stuff but all those jobs are not done in a day or two. Unfortunate.