Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Variable persistence for suspend-and-resume programs

by polettix (Vicar)
on Jul 23, 2005 at 21:00 UTC ( #477517=perlmeditation: print w/replies, xml ) Need Help??

Sometimes I make programs that could require some hours to run. This is usually to analyse some gigs of compressed data, or things like that, so I need to use them occasionally. The issue is that I would like to interrupt the process (e.g. to go home when I'm at work, or simply because I need 5 minutes of CPU) and let it resume without the need to restart it from the beginning.

Thus, I thought to program something that would support state preserving without too much hassle for the user (me :); of course, at the moment "state" means variable values, not other things. I started from vars and added some stuff to support this, here's what came out.

package vars::persistent; =head1 NAME vars::persistent - Perl pragma to predeclare persistent global variabl +e names =head1 SYNOPSIS use vars::persistent qw($frob @mung %seen); use vars::persistent '<file name.dat', qw($frob @mung %seen); vars::persistent::checkpoint('file name here'); # save variables vars::persistent::checkpoint(); # use last file na +me vars::persistent::restore('file name.dat'); # restore saved va +riables vars::persistent::restore(); # ditto, with last + file name =head1 DESCRIPTION This pragmatic module works much like L<vars>, from which the main imp +ort routine has been taken. In addition, each variable declared here is also track +ed, so that you can save them all to a file or retrieve their values, which can co +me handy if you're making an application which needs to save its state and retriev +e it later. See L<perlmodlib/Pragmatic Modules> and L<vars>. =head2 IMPORT You specify variables to I<use> much like the L<var> pragmatic module. + You cannot use sigils C<*> and C<&>, which aren't storable. You can optionally specify a filename prepending a C<<> to it. =head2 FUNCTIONS =cut use 5.006; use warnings::register; use strict qw(vars subs); use Storable; our $VERSION = '1.01'; my @variables; my $filename; sub import { my $callpack = caller; my ($pack, @imports) = @_; my $filename; my ($sym, $ch); foreach (@imports) { if (($ch, $sym) = /^([\$\@\%])(.+)/) { if ($sym =~ /\W/) { # time for a more-detailed check-up if ($sym =~ /^\w+[[{].*[]}]$/) { require Carp; Carp::croak( "Can't declare individual elements of hash or array" +); } elsif (warnings::enabled() and length($sym) == 1 and $sym !~ tr/a-zA-Z//) { warnings::warn("No support for built-in vars"); } ## end elsif (warnings::enabled(... elsif (($^H &= strict::bits('vars'))) { require Carp; Carp::croak( "'$_' is not a valid variable name under strict vars +"); } } ## end if ($sym =~ /\W/) $sym = "${callpack}::$sym" unless $sym =~ /::/; *$sym = ( $ch eq "\$" ? \$$sym : $ch eq "\@" ? \@$sym : $ch eq "\%" ? \%$sym : do { require Carp; Carp::croak("'$_' is not a valid variable name"); } ## end do ); push @variables, [$sym, $ch]; } ## end if (($ch, $sym) = /^([\$\@\%\*\&])(.+)/) elsif (($ch, $sym) = /^([\>\:])(.+)/) { if ($ch eq '>') { $filename = $sym } else { # Import routine inside... } } else { require Carp; Carp::croak("'$_' is not a valid variable name"); } } ## end foreach (@imports) restore($filename) if ($filename); } ## end sub import =over 5 =item my $ris = checkpoint($filename); Make a checkpoint, i.e. save specified variables to $filename. If omit +ted, last value specified in either C<checkpoint> or C<restore> (see below) is u +sed. Returns undef if file could not be opened, 1 otherwise. =cut sub checkpoint { my ($f) = @_; $filename = $f if defined $f; open my $fh, ">", $filename or return; foreach (@variables) { my ($sym, $sigil) = @$_; my $s = ( $sigil eq "\$" ? \$$sym : $sigil eq "\@" ? \@$sym : $sigil eq "\%" ? \%$sym : undef ); next unless $s; Storable::nstore_fd($s, $fh); } ## end foreach (@variables) close $fh; return 1; } ## end sub checkpoint =item my $ris = restore($filename); Restore variable values from $filename. If omitted, last value specified in either C<checkpoint> above or C<restore> is used. Returns undef if file could not be opened, 1 otherwise. =cut sub restore { my ($f) = @_; $filename = $f if defined $f; open my $fh, "<", $filename or return; foreach (@variables) { my $iref = Storable::fd_retrieve($fh); my ($sym, $sigil) = @$_; if ($sigil eq '$') { $$sym = $$iref } elsif ($sigil eq '@') { @$sym = @$iref } elsif ($sigil eq '%') { %$sym = %$iref } } ## end foreach (@variables) close $fh; return 1; } ## end sub restore 1; __END__ =back =head1 EXAMPLE Suppose you have a program that must do lengthy calculations. You may need your computer at some time, so you could like to stop the calcula +tions for a while, then restore them later - this module aims to help you do that seamlessly (more or less). # Yes, you'll get global variables... The following declaration will # preload values for variables if "file.dat" is present use vars::persistent '<file.dat' qw( $foo @bar %baz ); # Initialisation unless $foo is already defined, assuming it is if # the file is present unless (defined $foo) { $foo = 0; @bar = qw( follow me ); %baz = ( a => 10, b => [12, 35], c => { d => 'hallo', e => 'all'}) +; } # Install termination handler to save upon exit my $exit_please; $SIG{__TERM__} = sub { $exit_please = -1 unless $exit_please; exit(0) + } # Start long calculation that relies on above state variables # We assume that the single cycle can be taken to the end before # exiting here while (! $exit_please) { # Perform something here. Set $exit_please to 1 if calculation is # terminated. And yes, I don't want to care about race conditions # now... } # Save state variables for next time, if applicable vars::persistence::checkpoint() if $exit_please < 0; =head1 BUGS There are lots of bugs, but that's the real puzzle for you to solve - +where are they? I should learn how to export the two functions. No attempt to preserve integrity during save is attempted. I should re +ally save to a temporary, then move to the target file. =head1 AUTHOR I don't know who the author of is. Flavio Poletti E<lt>flavio@polettix.itE<gt> =head1 COPYRIGHT AND LICENSE A big chunk of this module is from the author(s) of I did not find their copyright notice, but I assume that it is the same as Perl itself being a CORE module. The rest of the stuff is Copyright (C) 2005 by Flavio Poletti For the same reason, I assume that the following should be quite fair: This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.6 or, at your option, any later version of Perl 5 you may have available. =cut
I also did a simple test script:
#!/usr/bin/perl use strict; use warnings; use Test::More 'no_plan'; ######################### # Three variables used as constants in the following my $rfoo = 10; my @rbar = qw( ciao a tutti ); my %rbaz = ( first => 'aaaa', second => ['bbbb'], third => { yes => 'a +nd so on'} ); # Declare three persistent variables. The mechanism is the # same as use vars. use vars::persistent qw( $foo @bar %baz ); # Initialise to the constant values above $foo = $rfoo; @bar = @rbar; %baz = %rbaz; # Silly verification here is($foo, $rfoo, "scalar value"); is_deeply(\@bar, \@rbar, "array"); is_deeply(\%baz, \%rbaz, "hash"); # Checkpoint, i.e. save to file vars::persistent::checkpoint('savefile'); # Reset values $foo = 0; @bar = (); %baz = (); # Restore from file vars::persistent::restore(); # file name is cached # These tests are the real one now is($foo, $rfoo, "scalar value"); is_deeply(\@bar, \@rbar, "array"); is_deeply(\%baz, \%rbaz, "hash");
I feel like I'm reinventing some wheel here, but it was not much work and, moreover, the most similar concept was maybe that of Session, which was a bit overkill here - I liked the idea to save and restore variables directly, instead of having some interface structure to work with.

I also know that there is much, much space for improvement, so I'd like to have some feedback from you wised monks :)

perl -ple'$_=reverse' <<<ti.xittelop@oivalf

Don't fool yourself.

Replies are listed 'Best First'.
Re: Variable persistence for suspend-and-resume programs
by Tanktalus (Canon) on Jul 24, 2005 at 04:22 UTC

    I don't want to take away from your solution at all - I'm sure I could find some uses for this idea at some point. But your two reasons wouldn't be among them.

    Going home from work: generally, I leave stuff running. In fact, I generally kick stuff off before I go home to let the computer keep doing my work for me. So maybe I'm missing a key component of why this is important. Perhaps it's because you need to watch it do its work, and if you're telnetted into a box, and then pack up your laptop, the connection is broken, sending a SIGHUP to your program. That's where VNC comes in - I run that in a VNC server window, and I can reconnect to it when I get back in in the morning.

    Suspending to get some CPU time back: that's what ctrl-z is for. And then "fg" to return it to the foreground.

    Unix isn't just an environment for running perl, y'know. ;-)

    Unix's motto is: Solve one problem, and solve it well. I just string a bunch of these together as needed. I can move about pretty much at will, connecting to servers with stuff running, and monitor, suspend, kill, disconnect, etc., with pretty much impunity. No special code required.

      I'm not in love with my solution, I consider it like a stone to "thump" on the door of the monastery of wisdom. Once I find what could solve the problem in the best and easy way, I'd probably throw it away :) This just to say that I feel quite positively prepared to criticism, and to thank you (and tlm) for the feedback.

      I have complete control over the CPU of my laptop. Moreover, most of the time I'm behind a proxy in an enterprise I work for as a consultant, so I wouldn't have occasion to control servers remotely in a comfortable way. This is why I do most of these calculations on my laptop - and why I'd like to arrive home and restart the process during the night :)

      Moreover, my current setup does not allow me to suspend and resume the laptop easily. This led me to try to code something: just because the particular issue could pop up in non-Unix world, or in situations where your suggestions could not be followed easily. In one word: when portability could be an issue (which I agree that is not something that happens every time :).

      perl -ple'$_=reverse' <<<ti.xittelop@oivalf

      Don't fool yourself.
      ++Tanktalus. Use unix job control here; that's what it's for. Instead of VNC, I recommend using screen -- you can set up a multi-window, multiprocess environment, but text-only, and detach/reattach from any ssh or telnet session. Within it, ctrl-z, bg/fg/jobs will be your friend. Good luck.
Re: Variable persistence for suspend-and-resume programs
by tlm (Prior) on Jul 24, 2005 at 00:11 UTC

    In the past, when I've needed this sort of thing, I've structured my code so that there's a well defined state object (which can be pretty complicated), and an "evolver" function that evolves the state object to some final condition. Then I simply use Storable to serialize and save the state object as needed.

    This approach may represent a bigger burden on the programmer than what you propose, but I trust it more. After all, this functionality is, almost by definition, useful only with long, i.e. precious, runs. When days' worth of computation are at stake, I prefer to take it upon myself to work out the details of what constistutes a state, and how this state is evolved, etc., rather than letting this analysis be somehow automated after-the-fact. If I were to automate it, I'd go whole-hog and figure out how to save all the memory image to disk and restore it later.

    BTW, I find checkpointing useful, but for reasons different from the ones you cite, e.g. suspending a job temporarily to give the CPU to something else. For these I just use kill -STOP/kill -CONT (yes, I'm a Unix creature). Also, GNU screen is invaluable for decoupling processes from login sessions. I often work on remote hosts, through connections that are not infinitely reliable; screen has saved my butt more often than I care to admit. (In fact, for all you Unix users out there, screen is one of those utilities, like Google, that once you start using it, you begin to wonder how anyone could live without it.)

    the lowliest monk

      I agree on the concerns only partially. Maybe an assembler programmer used to suspect of all those "high-level" languages, but today assembler is used only in a restricted portion of programmers world. What I mean is that I'm looking for some facility to portably handle my issue, which can of course be quite different from what I've coded above. To put it in another way, I'd like some way to factor out all the needs that a suspend-and-resume approach would have, in order to simplify the task the next time. Until then, I agree that it's something that I have to deal with with extreme care.

      I'm mostly a Linux guy, so the TSTP/CONT signaling would be fine most of the time. But sometimes my programs are used by less evolved guys, bound to use some legacy OS, which forces me to use it on that OS occasionally. The bottom line is that I was looking for something portable.

      Checkpointing would help here to save state at intermediate steps, of course. I consider this a nice side effect :)

      perl -ple'$_=reverse' <<<ti.xittelop@oivalf

      Don't fool yourself.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://477517]
Approved by Arunbear
Front-paged by planetscape
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (5)
As of 2022-11-29 01:08 GMT
Find Nodes?
    Voting Booth?