Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks, I am faced with a performance issue here. I am running the same perl script in two different servers having perl 5.8 installation. The script is simple and having only one File::Find operation in it. When I run the script, what I notice is that the process is hogging a lot of memmory in one server and very less in the other server. I have checked the File/Find.pm file in the installation directory and noticed that there is a difference in size between the two Find.pm files. Can some body please tell me what could be causing the difference in memmory utilisation in these two servers? Please enlighten me and revert back to me for any other details on the same.

Replies are listed 'Best First'.
Re: File::Find hogging memmory.
by atcroft (Abbot) on Oct 11, 2004 at 14:39 UTC

    My first suggestion would be to look at the Find.pm file for the $VERSION, to see which is newer (I would also compare versions of perl between the two machines in this respect as well-the current version of File::Find on CPAN appears to be 1.07, which is included with perl 5.8.5). If the one that uses less memory is the newer, then it may be related to an issue that was fixed between the versions. If it is the newer that is using more memory, then perhaps providing those version numbers here might trigger someone's recall of a similar issue, possibly with either what they found, or how they resolved it.

    Hope it helps (at least in providing you a starting point to work from).

      Rather than grovelling around in @INC to locate the Find.pm - it is probably easier to do:

      perl -MFile::Find -e' print File::Find->VERSION'
      or
      perl -MFile::Find -e' print $File::Find::VERSION'

      /J\

Re: File::Find hogging memmory.
by matija (Priest) on Oct 11, 2004 at 14:39 UTC
    You would have a much better chance of arriving at some illuminating answer if you ran diff -wu on the two versions of File::Find.

    Since you didn't give the exact versions of Perl and File::Find that is installed on each system, even the most diligent of the monks can't do it for you.

Re: File::Find hogging memmory.
by grinder (Bishop) on Oct 11, 2004 at 15:12 UTC
    what could be causing the difference in memmory utilisation in these two servers

    You don't specify what the wanted subroutine is doing, and whether the two servers have similar characteristics in terms of what you are looking for.

    If you are looking for files and pushing them onto an array, and on one server you have lots of matches and on the other you have very few then you have a good explanation as to why the process on one server is so much larger.

    If this is indeed the case, then you need to adopt a different strategy. One thing I have done in the past is to write the wanted files into a workfile. Then at the end of the find operation you can go back through the file and read the results. The more files you match, the more you save, since you only ever have to keep one record in memory.

    - another intruder with the mooring of the heat of the Perl

Re: File::Find hogging memmory.
by Joost (Canon) on Oct 11, 2004 at 14:50 UTC
    I can't really help you without more information:

    Which operating system(s) are you running the code on?

    Which perl versions are you using exactly? (use perl -V to find out)

    What is the code of your script?

    Also, are you sure you are using the same script on both machines?

    Do both machines have (about) the same amount of files in the directory tree, or does one have many more files than the other?

Re: File::Find hogging memmory.
by asifk1981 (Novice) on Oct 11, 2004 at 15:50 UTC
    Hi monks,
    Thanks a lot for the guidance!!!   I am here to provide more info regarding the issue.

    The version of perl in the servers are
    1) server where the process hogs more memmory - v5.8.0 built for PA-RISC2.0-thread-multi-LP64
    2) The other server - v5.8.0 built for PA-RISC1.1-thread-multi

    Both the servers are having the same OS HP-UX B.11.00. I have verified that the test script I run is the same on both machines, which is as follows:.

    #!/usr/local/bin/perl use File::Find; sub findtime { if ( -f "$File::Find::name" && "$File::Find::name" =~ /.*\.txt/i) { } } while (true) { sleep 3; find(\&findtime,"/test"); } and the number of files( both total and *.txt files) under the directo +ry /test is the same. Additionally, i have observed that the memmory +hogged by the server(which has the issue), apart from being huge at t +he initiation itself, grows with time. I have done a diff between the File/Find.pm files of the two servers a +nd noticed that following are the differences.( lets call the server +hogging memmory as s1 and the other as s2) s1 -- 17c17 < find(\&wanted, @directories_to_seach); --- s2 --- > find(\&wanted, @directories_to_search); ---------------------------------------------------------------------- +--- s1 -- 42c42 < order they are given. In essense, it works from the top down. --- s2 -- > order they are given. In essence, it works from the top down. ---------------------------------------------------------------------- +---- s1 -- 571c571 < local($dir, $name, $fullname, $prune); --- s2 -- > local($dir, $name, $fullname, $prune, $_); ---------------------------------------------------------------------- +----- s2 -- 702a703 > $_ = $name if $no_chdir;

    Apparently, the Find.pm in s1 looks like to have undergone some manipulations, by the root, which is highly unlikely. Please let me know if I can provide any further info, before reaching some conclusion.

    janitored by ybiC: Removed frowned-upon <pre> tags, minor format tweaks for legibility

      First of all, did you follow gellyfish's suggestion and compare the versions of the File::Find module itself? Did they report version differences?

      Secondly, while you included code to show how it is deciding to do something with the desired files, you showed nothing regarding what you are doing with the match. Are you storing something related to the match in an array, hash, object, or other "growable" data structure of some kind?

      My only other concern would be in that you are creating what is essentially a daemon process that runs continuously, with short (3s) pauses between processing. Not knowing your particular application, I cannot say this is or is not the best way to approach it, but if it is not essential to the application, you might consider limiting the number of times the app runs, in combination with the system's cron (or similar) feature, to keep the app from running too long if it does indeed have some form of memory leak.

      Not sure that helps, but hope it at least gives you something to look at.

        The two Find.pm versions are compared and they are found to be the same (1.4). Still when I do a manual diff between them I could see the differences, as mentionjed in the previous mail from me. Please not thet this Find opetations are not part of any application or any other processes. I have written some test scripts that have only one Find operation in it, and obserbved that one of the servers hoggs a lot of tmemmory for the process. Please share ur ideas...
Re: File::Find hogging memmory.
by asifk1981 (Novice) on Oct 13, 2004 at 05:22 UTC
    can some one please shed some light into the issue. I have given more details on the issue in the above mail. It is a bit urgent. Please enlighten me!!!