This is by far the simplest yet most terrifying Perl solution that I have ever developed. I have written entire mod_perl websites and associated toolsets used by several fortune 100 companies but I can't bring myself to implement this on one of my own personal systems.

This system is booted (/boot, /, /var, /usr, etc...) off of an OCZ PCI-Express SSD card. Unfortunately every 1-3 months the card locks up resulting in I/O errors. In this case, even if I have an open shell on the system I am only able to run programs that are currently in disk cache in memory. Attempting to exec all other processes will result in I/O errors until a reset. There are usually no direct hands near the box and unfortunately I do not have access to out of band management of any kind to reset the box (IPMI, BMC, iLO, RSA, DRAC, PDU, eRIC, ATEN IP8000, etc...). I feel that my only option is to attempt to read a few KB from one of the ~4K files in /usr/bin and do an immediate (or postponed) reset if I get an EIO error back from sysread. I need to be able to reset the box immediately if I cannot read from one of the logical volumes on the SSD.

Keep in mind this is a very preliminary rough draft and I do not take posting to perlmonks lightly. However, I would appreciate any input on the logic.

• Specific Concerns
  • Is it possible that the "next" statement after the open statement could result in going to the start of the loop in the case of an I/O error? My gut says that this will be cached in the VFS and there are no errors from open for I/O that I could find in the open(2) man page.
  • I have *NO* way to test this script/daemon. How can one emulate EIO?
  • This is a desperate situation. I cannot afford to replace the SSD with a proper RAID-1 but I do have good backups.

#!/usr/bin/perl

use strict;

use Errno;

# 3740 files, 3739 max index
my @files = glob("/usr/bin/*");
my $range = $#files; #scalar(@files)-1;

while (1) {
    my $random = int(rand($range));
    my $filename = $files[$random];
    my $buffer;

    print STDERR "Checking $filename - ( $random / $range )\n";

    #my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$c
+time,$blksize,$blocks) = stat($filename);

    open(FILE, "< $filename") or next;

    my $retval = sysread(FILE, $buffer, 8192);

    if (!defined $retval) {
        if ($!{EIO}) {
            print STDERR "I/O Error on $filename. Resetting.\n";

            #system("/var/tmpfs/reboot -nf"); -- MUST BE IN MEMORY FS 
+AND STATICALLY LINKED. system calls /bin/sh but should be cached in m
+emory
            #exec {'/var/tmpfs/reboot'} '-nf'; -- MUST BE IN MEMORY FS
+ AND STATICALLY LINKED
        }
    }

    close(FILE);

    sleep(15);
}
[download]

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.