I've had a somewhat related incident happen recently. I have a Plex system, with a 4TB SSD that has about 2.5 TB of library data.

Every night, this data is rsync'd to replica system at another one of my properties (with the --delete-after flag set).

One day, Plex wasn't working. Checked, and the disk had gone away in much the same way yours did. However, the /media/STORAGE directory still existed, but was empty because it didn't have the SSD mounted, so my system blindly just erased everything on the remote backup replica.

Thankfully, I also have an online storage that uses an rsync push for all of the data I back up (including the Plex libraries), which I don't use the --delete-after flag for, specifically because of these hiccups.

Took a bit of time to restore nearly 3TB of data from an online source, but thankfully I was able to recover everything (helps to have a 1Gbps fibre connection).

So just as a precautionary note, even if you have a good backup regimen, it pays to review it to ensure that it isn't set up in a way where if a disk goes away, it can't erase the backup.

Update: Now my backup script checks to see if a file under the /media/STORAGE directory exists before proceeding. If not, it alerts me.

In reply to Re: [OT] Reminder: SSDs die silently by stevieb
in thread [OT] Reminder: SSDs die silently by afoken

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.