I'll keep you updated if we could make sure it's the "store" from different processes

I very rarely gamble, but I'll bet that this is your problem.

My experience of writing inter-process communication software with shared storage using Storable says that you are having collisions that result in broken writes.

You need to provide adequate exclusive locking mechanisms, which isn't easy.

You're running a CRM. That, by default means you may have multiple entries at once, all the time. You're likely only going to grow, so patching this with locks is not sustainable nor is it scalable.

Rewrite all of it to use a proper database. Doing anything else is a temporary band-aid and will only cost your company money for nothing.

Having a (potentially growing) multi-user input all using the same single output file is like pouring an ever increasing amount of liquid into a funnel and expecting the funnel to allow the increasing amount of liquid to flow through. It won't, no matter what you do.

Even if you successfully manage to set up a proper locking mechanism (trust me, this is hard), there's always the single-file contention. You will forever have problems no matter what.

To further, Storable is a proprietary format. Don't use it for the type of storage you're using it for. Serialize your data in a standard format.

I want to expand on this, just so my point gets across...

The general process of locking something for exclusive write is this:

my $data = $thing->fetch;

$thing->lock;

$thing->write('new data');

$thing->unlock;
[download]

...all well and good. However, these are ADVISORY locks, not actual disk-based locks on the file. That means a script that isn't taught to lock/unlock the file could just write all over it even if your fixed script has it locked. In other words, all of your software that deals with this file must all honour and set locks properly. Let's now not forget about the OS itself. Someone opens the file in Finder or Windows Explorer or something, that has no idea nor does it care about your software advisory locks. It'll write to your file no matter what.

Database. It's your only feasible way forward.

Update: Here is a snippet of an example of the only way I've ever found to 100% ensure that write locking would be effective... a database transaction, which rolls back if the write failed. Note the FOR UPDATE. (I'm no DB expert so I'm putting this out there for feedback on improvements while providing an example for the OP/thread):

    if ($bcast_id) {
       my $broadcast_transaction_status = eval {

            # We have to disable AutoCommit so that all of the DB task
+s
            # are build into a single transaction. This locks the row.

            $self->dbh->{AutoCommit} = 0;
            $self->dbh->do('BEGIN');

            # We need to ensure the broadcast is still available. If i
+t is,
            # we claim it. If it isn't, we do nothing

            my $bcast_status_check_query = qq~
                SELECT BroadcastID 
                FROM $db_table
                WHERE BroadcastID=?
                AND HostRunning IS NULL
                FOR UPDATE
            ~;

            my $broadcast_unclaimed = $self->dbh->selectrow_array(
                $bcast_status_check_query,
                undef,
                $bcast_id
            );

            if ($broadcast_unclaimed) {
                # We only update the DB table with broadcast claimed s
+tatus
                # if nobody else has claimed it yet

                $self->dbh->do(
                    qq~
                        UPDATE $db_table
                        SET HostRunning=?
                        WHERE BroadcastID=?
                        AND HostRunning IS NULL;
                    ~,
                    undef,
                    "$hostname, Process $$",
                    $bcast_id
                );
            }
            else {
                # If the broadcast was claimed in between our first ch
+eck and our
                # second check inside the transaction, we set this bro
+adcast to no
                # longer available

                $bcast_id = 0;
            }

            # Commit the transaction and re-enable AutoCommit so that
            # it doesn't impact other DB operations

            $self->dbh->commit;
            $self->dbh->{AutoCommit} = 1;

            1;
        }
    }

    if (! $broadcast_transaction_status) {
        $self->dbh->rollback;
        $bcast_id = 0;

        _email_sysadmins(
            $self->dbh,
            "Broadcast $bcast_id claim rolled back",
            "Broadcast $bcast_id host claim transaction rolled back: $
+@"
        );
    }
[download]

Note the likelihood that the code may have come from a CMS ;) What happens there is we check a row, make sure nobody else has updated it, proceed to update it, but if the update fails, the entire transaction is rolled out with the DB not being touched at all. This is true write locking which can not be had if using a file as the backend.

Its called rename an atomic operation

I like my bikeshed red, but I'm red/green colourblind (honestly), so I don't understand what you're referring to in your comment.