snam has asked for the wisdom of the Perl Monks concerning the following question:

Hello, perl programmers.
foreach my $d(readdir(PROC)){ next if $d !~ /^[0-9]+$/; my $procdir = "/proc/$d"; open(IN, "<$procdir/status") || die "can't open status with$pr +ocdir"; while(<IN>){ @a = split(/\s+/, $_) if /Uid/; $temp[0] = $a[1]; @b = split(/\s+/, $_) if /VmSize/; $temp[1] = $b[1]; }
what is the fastest way to store a file into an array or a variable? because in my code, I open a directory and find a file that I want to open, and store the name of the file into the variable $d, but by the time when I try to open that file, that file is already gone. so i get an error message saying it can't open the file. Is there anyway to prevent this? like as soon as it sees the file name, open it and store it in a variable to prevent the error? Thanks.

Replies are listed 'Best First'.
Re: fastest way to open a file and stroing it?
by jepri (Parson) on Feb 03, 2003 at 04:55 UTC
    Regrettably no. You do have to do that directory call first. This is a race condition that is common on multiuser systems.

    The best you can do is either change the kernel to accomodate your desires (you could probably write a module) or you could implement that bit of code in C to reduce the amount of time you linger.

    In perl, I would rewrite your code so that you open all the files first, then read from them one by one. In most unices, a file won't 'disappear' until the last file handle is closed.

    foreach my $d(readdir(PROC)){ next if $d !~ /^[0-9]+$/; my $procdir = "/proc/$d"; open($filehandles{$procdir}, "<", $procdir/status") || warn "c +an't open status with$procdir"; } foreach my $fh ( keys %filehandles ) { while(<$fh>){ @a = split(/\s+/, $_) if /Uid/; $temp[0] = $a[1]; @b = split(/\s+/, $_) if /VmSize/; $temp[1] = $b[1]; } close $fh; }

    ____________________
    Jeremy
    I didn't believe in evil until I dated it.

Advisory locking to 'lock' a directory
by Coruscate (Sexton) on Feb 03, 2003 at 05:24 UTC

    The first thing that comes to mind for me is 'locking' the directory. Do you get this error quite often? If this is happening because the files in this directory are changing quite rapidly, then I would suggest doing a lock of some kind on the directory while you fiddle around with it.

    Note that this method only works if any and all programs that modify files in this directory follow this locking mechanism. For example, if this were a directory that users FTP files to and can delete files (from FTP) from this directory, this method simply would not work as desired because the users' FTP clients are not going to know that your lock exists, let alone how to go about gaining this lock. Having said that, here's how you'd do it:

    Use a semaphore file to 'lock' the directory. Note the quotes around the word 'lock', because this does not deny any program access to the directory. If a program wants to walk in and change files, it is free to do so. The thought behind using a semaphore file is this: every single time you plan on modifying a file within the directory, you open a file for writing, lock it with Perl flock() command and then go about your business. Example code (untested):

    #!/usr/bin/perl -w use strict; use Fcntl ':flock'; my $dir_path = '/home/snam/subdir'; my $lock_file = $dir_path . '/lock.lck'; # Get a 'directory lock' (note the quotes again) :) open LOCK, '>', $lock_file or die "Could not lock directory '$dir_path': $!"; flock LOCK, LOCK_EX; # Now do whatever you want with this directory. # If all programs use the locking mechanism, they # won't have access to this directory until you execute # a line that says "close LOCK;" (or when the script exits) opendir DIR, $dir_path or die "Could not open directory '$dir_path': $!"; my @files = readdir DIR; closedir DIR; for my $d (@files) { next if $d !~ /^[0-9]+$/; my $procdir = "/proc/$d"; open IN, '<', $procdir . '/status' or die "Can't open status with $procdir: $!"; while (<IN>) { # Fill this in yourself... } } # We're done with the directory! Let someone else access it close LOCK;

    Update: As soon as I posted, I realized I left out the actual flock() statement to place an exclusive lock on the semaphore file {grin}


          C:\>shutdown -s
          >> Could not shut down computer:
          >> Microsoft is logged in remotely.
        

      And you think the kernel is going to pay attention to any locks a user program might have? We are talking about the /proc file system here.

      The content of the /proc filesystem rapidly changes, and there isn't much you can do about it. Remember that /proc is nothing more than some hooks inside the kernel. Processes come and go, and so do their associated files in /proc.

      Abigail

Re: fastest way to open a file and storing it?
by Zaxo (Archbishop) on Feb 03, 2003 at 06:00 UTC

    This is a case where you may want to go to the system. The ps system call will take a snapshot of the processes on the system. The ephemeral nature of procfs does not interfere, as it does in your method. The ps call can have its output tailored to give what you want. Check your local 'man ps' to see if the options match in your version of ps.

    my $psout = `/bin/ps -A h o uid,vsize`; my @procs = map {[split]} split "\n", $psout; # To get total vm usage by uid, my %vm_usage; $vm_usage{$_->[0]} += $_->[1] for @procs;

    After Compline,
    Zaxo

Re: fastest way to open a file and stroing it?
by pg (Canon) on Feb 03, 2003 at 05:36 UTC
    Don't focus on improving the performance of your program, as a solution for this. I am not saying you should not improving the performance of your program, but I am saying it is not a solution for this problem.

    The point is that, whatever how fast your porgram is, there is still a chance that it is not fast enough for you to catch the file EVERY TIME, before it is gone.

    I always believe that a solution has to a solution, not something that improves the chance.

    I think you have to look at what you are trying to do again, and resolve this as a whole. Never waste your enegy on things, for sure, will not work all the time.

    Synchronization is the way to go. However I don't know your entire story, and I cannot really help in the details, but thru your capable hands, nothing is unresolvable, but make sure go straight, don't be side tracked.
      Thx, pg^^. Just like you said, I gave up on trying to improve the performance of my program, but instead, I checked whether the file that I stored in the variable exists or not by using "-e $filename". What you wrote was really helpful. Thank you again. snam
Re: fastest way to open a file and stroing it?
by BrowserUk (Patriarch) on Feb 03, 2003 at 05:40 UTC
    It might be possible to use (something like) chmod 0nnn, <path/*>; to set the files read-only--see your local man pages for the correct values of nnn to use--to whatever group/userid the process that is deleting them runs as, and then reset the permissions once you've processed them. However, that would probably mean that you would have to delete them in your code and could cause errors in the other process. You might also get away with creating a symbolic link (terminology?) to reference them (using a system utility?) that would keep data around until you deleted the link.

    I'm really out of my knowledge base on *nix, assuming that is your OS, so take this with a big pinch of salt, but the ideas may be useful.

    UpdateAnd didn't I prove it! I completely missed any association between the PROC arg to readdir and anything to do with the system process list. D'oh!


    Examine what is said, not who speaks.

    The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.