Re: Perl Memory problem ... (Memory Tools References)
by eyepopslikeamosquito (Archbishop) on May 10, 2020 at 05:49 UTC
|
If you give us more information we should be able to help.
Meanwhile, these old nodes might give you a clue as to what your problem might be.
Using this node for Memory Tools References nowadays:
General Areas to consider when faced with a Memory Leak
- Stability: Once your process grows to a certain size, does it stay there, or does it grow further over time?
- Large CPAN Modules. Are you using any?
- Circular references. See Eliminate circular reference memory leak using weaken (Perl Maven).
- Scoping.
- File/Database handling.
- Tools. Try one or more from the following section.
Perl Tools and Advice
- Profiling Memory Usage talk by Tim Bunce (YAPC::NA Austin Texas 2013 featuring Devel::SizeMe). Resident size vs vm size, rss, segments, /proc/pid/maps (e.g. SO maps question), malloc and the heap, perl internals, pads, constant folding, arenas, recursion, perl data structures (lots of pointers), ... (update: this reply describes memory used by lexical variables in subroutines at the 09:20 minute mark)
- Plans for adding memory profiling to Devel::NYTProf
- Devel::NYTProf
- Devel::Gladiator
- Devel::LeakTrace
- Test::LeakTrace
- Devel::Leak
- Devel::FindRef
- Devel::Cycle
- Test::Memory::Cycle
- Devel::MemoryTrace::Light
- Mini-Tutorial: Perl's Memory Management by ikegami
- As pointed out by dave_the_m: "once the perl interpreter has exhausted all memory, there's very little it can do. There is a reserved fixed buffer to allow it to display the OOM error itself, but doing anything complex like displaying a Carp::confess()-like stack backtrace would require many memory allocations to generate the display, all of which would themselves fail".
General Tools
- Clang (Clang is a compiler front end for the C, C++, Objective-C, and Objective-C++ programming languages ... includes a static analyzer, and several code analysis tools ... Clang operates in tandem with the LLVM compiler back end)
- LLVM (LLVM is a set of compiler and toolchain technologies that can be used to develop a front end for any programming language and a back end for any instruction set architecture ... designed around a language-independent intermediate representation (IR) that serves as a portable, high-level assembly language that can be optimized with a variety of transformations over multiple passes)
- Boehm garbage collector aka Boehm–Demers–Weiser garbage collector (wikipedia)
Static Code Analysis
Code Profiling and Optimization
Testing Concurrent Software
Security Testing
Some Related PM Nodes
More Nodes Added Later
See Also
Updated: Many extra references were added long after the original reply was made.
| [reply] [d/l] [select] |
Re: Perl Memory problem ...
by davido (Cardinal) on May 10, 2020 at 16:04 UTC
|
Perl won't release memory back to the operating system at runtime, only on global destruction as the script terminates. However, Perl will re-use the memory that it has allocated and that it is no longer using. For example, if you declare a lexical variable in one scope and load it up with 30MB of data, but then let that variable fall out of scope, Perl should be able to re-use that memory space.
Places to look:
- Circular references: These can cause memory leaks since the memory never has its reference count drop to zero, Perl can't re-claim it for internal use.
- Modules: Are you using half of CPAN? There's nothing wrong with using Moose, DBIx::Class, and many of the other slightly heavy distributions on CPAN, but they do use memory.
- Scoping: Are you allowing lexical variables to fall out of scope? (You should be.) Program with the objective of keeping state as ephemeral as possible; let variables have the narrowest possible scope - particularly those that are holding large chunks.
- File/Database handling: Are you slurping in large files or large Database query data sets? Process smaller chunks at a time.
- Stability: Once your process grows to 30MB, does it stay there, or does it grow further over time? Frankly, even if it's consuming 100MB, that's not the end of the world as long as you can count on it staying close to that plateau. If it keeps on growing over time, you've got a bigger problem (a memory leak). A well-behaved process that hits a memory plateau and stays there is usually acceptable.
- Tools: Try valgrind, Devel::LeakTrace, Devel::MemoryTrace::Lite, and other such tools can help you to watch how memory is growing and being used.
| [reply] |
|
| [reply] [d/l] [select] |
|
Thanks. I was probably misremembering Tim Bunce's YAPC talk: https://www.youtube.com/watch?v=GIIeOntmojg. I'll have to review it again when I have a moment. Your research looks thorough, though.
Update: At about 4:10 Tim discusses malloc behavior, and suggests that larger allocations are mmapped, and the gist seems to be that while rare for malloc to actually give back, it is more likely for those larger mmapped chunks. So, thanks for motivating me to find that talk again. I'll have to watch the rest of it again this evening. It does confirm your assertion, too.
| [reply] [d/l] |
Re: Perl Memory problem ...
by haukex (Archbishop) on May 09, 2020 at 22:11 UTC
|
That's not much information to go on, and nowadays, 30MB isn't that much at all. Perhaps you could show some code to reproduce the issue (SSCCE, How do I post a question effectively?), and explain what the problem is? Like for example, if the memory use continually goes up, that would indicate a memory leak.
| [reply] |
Re: Perl Memory problem ...
by Corion (Patriarch) on May 09, 2020 at 20:49 UTC
|
Somewhere, you're keeping memory alive.
Without seeing your code, it is impossible for us to know how to change your program.
Either reduce your code and post a short, self-contained example, or consider how you can periodically restart your chat server so all accumulated memory gets reclaimed.
| [reply] |
Re: Perl Memory problem ...
by perlfan (Vicar) on May 12, 2020 at 02:36 UTC
|
Need to see what you're doing. You don't need to release memory to the OS if you've scoped things properly. Another suggestion, don't import everything in your use statements. Idk what you did here, but I can image a list of imported modules a mile long. That or you used Moose or Moo and still included the kitchen sink. | [reply] [d/l] |
Re: Perl Memory problem ... (circular references)
by Anonymous Monk on May 10, 2020 at 04:36 UTC
|
| [reply] |
Re: Perl Memory problem ...
by Sascha2018 (Acolyte) on May 13, 2020 at 17:17 UTC
|
Hello.
Thanks for the replies.
Here is a little bit of my perl script i wrote.
The problem is: At start time it uses 10MB Memory.
And if a user writes a message ( SENDMESSAGE ) , it needs for every message ( 300 KB of memory ).
Now ... how can i fix the problem?
use feature 'state';
use Time::HiRes;
require "config.cfg";
require "Subs.pl";
require "format.pl";
require "server/mainprogram_subs.pl";
require "ConnectDB.pl";
require "loadconfig.pl";
require "$config{language}.lng";
require 5.002;
use IO::Socket::SSL 'inet4';
$IO::Socket::SSL::DEBUG = 3;
use IO::Select;
use Time::HiRes;
use HTML::Template;
use Text::Wrap;
use Fcntl ':flock';
use strict;
no strict 'refs';
later in my program:
if($buffer =~ /^SENDMESSAGE$digest\|(.*?)$/){
my $what = $1;
my($room,$touser,$kind,$message,$fromuser)=
+split /\|/, $what;
my $myuser = $fromuser;
my $update_sock = getCurrentIdent($myuser);
if( SockExists($update_sock) ) {
$clients{$update_sock}->{lastsrvpost} = ti
+me;
}
if($touser eq "" && $room ne "" && $kind ne
+"PRVMESSAGE") { sendToSocket($message, { toroom => $room, kind =>
+ $kind, fromuser => $myuser } );
}elsif($touser ne "" && $room eq "" && $kind
+ ne "PRVMESSAGE" ) { sendToSocket($message,{ fromuser => $myuser, tou
+ser => $touser, kind => $kind } );
}elsif($touser eq "" && $room eq "" && $kind
+ ne "PRVMESSAGE") { sendToSocket($message, {});
}
next;
and the sendToSocket routine is the following:
sub sendToSocket {
my( $data, $params ) = @_;
if( $data eq "" ) {
return 0;
}
if( $params->{toroom} ne "" && $params->{touser} eq "" ) {
foreach my $ident( keys %clients ) {
our $nick = $clients{$ident}->{nick};
if( isInIgnorelist($nick, $params->{fromuser}) && $co
+nfig{ignore_modus}{$params->{kind}}==1) {
}else{
my %info = loadMemberInfo($nick);
if( lc $params->{toroom} eq lc $info{room}) {
my $fh = $clients{$ident}->{sock};
if( defined $fh ){
my $msg = createFilter($ident, $data);
$fh->syswrite($msg . "\n");
undef $fh;
undef $msg;
undef %info;
undef $nick;
}
}
}
}
}elsif( $params->{toroom} eq "" && $params->{touser} ne "" ) {
my $user = $params->{touser};
my $sock = getCurrentSocket($user);
my $sn= $sock->{sockname};
my $socket = $sock->{socket};
my $ident = getCurrentIdent($user);
if( isInIgnorelist($clients{$sn}->{nick}, $params->{from
+user}) && $config{ignore_modus}{$params->{kind}}==1 ) {
}else{
if( defined $socket ){
$msg = createFilter($ident, $data);
$socket->syswrite($msg . "\n");
undef $msg;
undef $ident;
undef $sock;
}
}
}elsif( $params->{touser} eq "" && $params->{toroom} eq "" ) {
foreach my $ident( keys %clients ) {
my $sock = $clients{$ident}->{sock};
$msg = createFilter($ident, $data);
if( defined $sock ){
$sock->syswrite($msg . "\n");
}
undef $msg;
undef $sock;
}
}
}
But if i use undef, nothing happens. It still uses the memory :(
Hope you can help.
Thanks
Regards Sascha | [reply] [d/l] [select] |
|
Most likely, your problem lies somewhere else as here you clean out the variables you create locally.
Take a look at Devel::Cycle and/or Devel::LeakTrace and Devel::FindRef to see where your program is collecting memory. Ideally, you can eliminate all server/client interaction and use a faked client message in a loop to reproduce the problem.
You might want to change the coding style from
our $nick = $clients{$ident}->{nick};
... to
my $nick = $clients{$ident}->{nick};
| [reply] [d/l] [select] |
Re: Perl Memory problem ...
by Anonymous Monk on May 11, 2020 at 03:09 UTC
|
Remember that operating systems are designed to be lazy. If there's no pressure on the memory resource, it really doesn't care to reduce the exact amount that you are now using. Should pressure later develop, it will try more aggressively to determine which portions of your memory are now being used and which can be safely stolen for other purposes. But, until that actually occurs, it won't bother. | [reply] |
A reply falls below the community's threshold of quality. You may see it by logging in. |