Re: Merge log files causing Out of Memory

Your code is very strangely indented, quite hard to read. You may want to use perltidy and/or a better editor.
It actually looks like someone ported a shell script, sprinkled some parts of a C program over it, and copied some pieces of ancient perl scripts of dubious origin. Yes, perl will understand what you want, but it does not look very perlish and is not very efficient.
Your code uses lots of global variables where a local scope (see my) is sufficient.
use strict; and use warnings; are missing
$profile = `\. ~/.profile`; # Load user profile does not do what the comment says. And it is most likely completely useless.
eval is useless in $log_for_days = eval(86400 * 91);, in $dcnt=eval(@data-1); and all other places.
&date_calc; and &get_dirs; is Perl4-style. Avoid the ampersand <UPDATE>when calling functions</UPDATE>, it usually does not do what you intent to do.
Your code use BAREWORD file handles and the insecure two-argument open, better use the three-argument form and scalar file handles. Also use or instead of || here. And use $! instead of $^E: open my $dirfile,'<','/opt/scripts/dirs') or die "Can't open /opt/scripts/dirs: $!";
Neither the i flag nor the g flag are needed for matching # in the lines read from /opt/scripts/dirs.
get_dirs could be implemented using grep to both find unique directory names and to remove comments. Returning right after reading the file's contents automatically closes the file handle IF you use a scalar file handle: return grep { !/\#/ && !$seen{$_}++ } <$dirfile>;
Using backticks (``) introduces a lot of security problems, especially if you run the script as root. Shells don't always parse data pasted into backticks as you intent. Try to use perl functions instead: `rm -f ...` should be replaced by unlink or by the remove_tree() function from File::Path. `cat ...` should be replaced by open, readline alias <$handle>, close. `ls` should be replaced by opendir, readdir, closedir, or maybe glob.
@array=`cat ...` reads the entire file into memory. If your log files are large, you may quite easily run out of memory. After that, you iterate over the array line by line. Better use open, as explained above, and use a while loop to read the file line by line. This way, you have only one line in memory, even if your log file contains terabytes of data:
```
open my $handle,'<',$filename or die "Can't open filename: $!";
while (my $line=<$handle>) {
    chomp $line;
    if ($line is interesting ...) {
        # do something with $line ...
    }
}
close $handle;
[download]
```
@encdir=`ls .... 2>&1` is even worse than just @encdir=`ls ...`, you end with error messages as directory names in @encdir. You REALLY don't want that, use opendir/readdir/closedir or glob, as explained above.
You are doing really weird things with arrays and strings, stringifying arrays just to split them back into arrays. Avoid that. Perl functions can return arrays, don't pass function results around by writing to global variables.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Comment on Re: Merge log files causing Out of Memory Select or Download Code

Replies are listed 'Best First'.
Re^2: Merge log files causing Out of Memory (just a note on ampersand) by Discipulus (Canon) on Aug 25, 2016 at 10:26 UTC
as always you are very complete and competent, but (as other times) i must dissent on: &date_calc; and &get_dirs; is Perl4-style. Avoid the ampersand, it usually does not do what you intent to do. is not Perl4, is perfectly valid Perl5, and is reffered by official docs as `optional in modern perl` It is still `not` optional in three cases: `# while naming a sub like in: defined &my_sub_name; # doing indirect sub call (but $subref->() is another valid option) &$subref(); # making a reference to a sub $coderef = \&handler;` [download] ..it usually does not do what you intent to do. just means that `&get_dirs` receive the current `@_` even if no args are specified. The programmer must be aware of this and the feature can be also used in a profitable way. The example in perlsub is exahustive: `&foo(1,2,3); # pass three arguments foo(1,2,3); # the same foo(); # pass a null list &foo(); # the same &foo; # foo() get current args, like foo(@_) !! foo; # like foo() IFF sub foo predeclared, else "foo"` [download] L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re^3: Merge log files causing Out of Memory (just a note on ampersand) by hippo (Archbishop) on Aug 25, 2016 at 10:57 UTC
is not Perl4, is perfectly valid Perl5 Yes, it is perfectly valid Perl5. However it is also perfectly valid Perl4 and as afoken points out this chimes with all the rest of the script looking like Perl4 too. I don't have a perl4 installation to hand to test this on but would not be surprised if that's what the script is (or at least how it started out). Ampersand notation without good reason (such as the exceptions you have correctly quoted from the docs) is very much Perl4-style. Edited for typo and clarity	[reply]
Re^4: Merge log files causing Out of Memory (just a note on ampersand) by Discipulus (Canon) on Aug 25, 2016 at 11:16 UTC
Probably is only one of my vicious behaviours but im used to use `&function($arg);` to call function loacally defined in the current script (file) to distinguish them from methods-functions imported by modules i use. L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re^3: Merge log files causing Out of Memory (just a note on ampersand) by afoken (Chancellor) on Aug 26, 2016 at 06:20 UTC
I've updated my posting to limit the "avoid the ampersand" rule to function calls. just means that &get_dirs receive the current @_ even if no args are specified Exactly for this reason. `&function;` has this surprising behaviour. The programmer must be aware of this Correct. The programmer must be aware of this, because it is nastily surprising behaviour due to backwards compatibility with Perl 4. In other words: Don't expect a novice to understand what happens here. Compare to what we tell our little children: "Don't touch the knife." ("Don't use ampersands in function calls.") You don't explain an average three-year old the physics of cutting, the medical problems of infections and loss of large amounts of blood, and so on. Later in the life, you or someone else explains the child (or the novice) why knifes (or ampersands in front of function calls) are dangerous, and how knifes (or ampersands in front of function calls) can be useful. i must dissent on: &date_calc; and &get_dirs; is Perl4-style. Avoid the ampersand, it usually does not do what you intent to do. is not Perl4, is perfectly valid Perl5 I never said that `&get_dirs;` was Perl 4, and I never said it was illegal in Perl 5. I named it Perl4-style, because that's how functions had to be called in Perl 4. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l] [select]
Re^3: Merge log files causing Out of Memory (just a note on ampersand) by choroba (Cardinal) on Aug 25, 2016 at 20:39 UTC
There's one more context where `&` isn't optional: `goto &func;` [download] See goto. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^2: Merge log files causing Out of Memory by malokam (Novice) on Aug 25, 2016 at 16:10 UTC
Thank you very much for the response. I really do appreciate the inputs. Obviously, I am newbie to PERL . This is a script that I created from looking at 3-4 existing scripts in place (read 10 years or above) and picking parts that would suit my needs. I will go through each and every point of yours and try to update it.	[reply]