Benchmarking with Memory Profiling

Arguile has asked for the wisdom of the Perl Monks concerning the following question:

Brethren,

I have recently begun the journey to CS enlightenment, with all its algorithm efficiency, holy "O" notation, and other joyous wonders¹. I have encountered a wall of thorns along this path.

At the moment I'm working with sort functions that need to read data from the disk, sort, then store back to the disk. Various constraints and data sets are provided, sometimes you have to have fit a sort into $i Mb or have it run under $j seconds². Prior to this I was happily using Benchmark, however the new memory considerations preclude its use as the sole tool. Checking CPAN's Devel and other packages turned up nothing that dealt with memory profiling. After reading a bit more into Perl’s memory management, reference counting, garbage collection, and general recycling... it became apparent how naļve I was in this respect. A straight comparison inside the same runtime interpreter would have produced incredibly skewed results anyways.

This leaves me where I am now, concerned with relative performance in memory, speed, and scratch space between scripts yet with no overall way to compare. I'd guess some type of fork() call w/ IPC or an external framework are needed but I'm only beginning programming so these are, for the moment, well beyond my ability to write.

I ask unto ye, are there readily available tools for doing these comparisons?

-- Brother Arguile

¹ I realise Perl is not considered by some to be the ideal language to learn this in, but I have to balance my immediate professional needs with the time consuming theoretical material. It simply doesn't make sense to jump into Scheme or C/C++ full force at this time.

² At this point I'm not so concerned with absolute limits as relative performance.

Comment on Benchmarking with Memory Profiling Select or Download Code

Replies are listed 'Best First'.
Re: Benchmarking with Memory Profiling by andreychek (Parson) on Jun 10, 2001 at 07:46 UTC
You're in luck! There are all sorts of methods to benchmark code. I found that there are some excellent explanations on benchmarking in the mod_perl guide. While the mod_perl guide does talk about mod_perl, much of the information there is relevant to all Perl uses. The first example of benchmarking they give uses the Benchmark module, which you already tried. However, they give other examples using modules like Time::HiRes and GTop. Here is an example of the code you would use with Time::HiRes (copied from the mod_perl guide): `use Time::HiRes qw(gettimeofday tv_interval); my $start_time = [ gettimeofday ]; sub_that_takes_a_teeny_bit_of_time(); my $end_time = [ gettimeofday ]; my $elapsed = tv_interval($start_time,$end_time); print "The sub took $elapsed seconds."` [download] But the good, juicy information doesn't stop there. They get into code profiling, memory usage, and other goodies -- including the speed differences between method calls vs function calls, along with offering various advice on good coding habits to reduce memory usage. Feel free to skip any sections that are mod_perl/web specific, if that isn't what you intend to do. Again though, much of what they offer there goes for any code, not just mod_perl and web applications. -Eric	[reply] [d/l]
Re: Benchmarking with Memory Profiling by ZZamboni (Curate) on Jun 10, 2001 at 07:17 UTC
As I told you in the CB I don't have an answer to this , but I'd like to comment on this: I realise Perl is not considered by some to be the ideal language to learn this in I don't think this is necessarily true when studying algorithms. When I was taking my algorithms class, I often found that writing a quick perl program was the easiest way of testing an algorithm for a homework, for example. Perl makes it very easy to write things almost in pseudocode, and it has plenty of high-level operations, so that translating things from an algorithm in the book to working code can be relatively painless. And with tools like Benchmark, at least time measuring is easy. --ZZamboni	[reply]
Re: Re: Benchmarking with Memory Profiling by Arguile (Hermit) on Jun 10, 2001 at 07:40 UTC
I agree with the pseudo coding, often I can just pull the logic example right out of the book and have a working version to play with. I also find it's allowing me to more easily grasp the root concept and not get caught up in the perculiarities of a specific machine/platform as seen in many of the C/C++ answers in the books. I only put that in there because I asked a like question in a different forum and got all sorts of "use C++" or "use LISP" answers.	[reply]
Re: Re: Re: Benchmarking with Memory Profiling by ZZamboni (Curate) on Jun 10, 2001 at 07:47 UTC
I only put that in there because I asked a like question in a different forum and got all sorts of "use C++" or "use LISP" answers. And that's why I qualified "when studying algorithms". Because there are some things for which other languages may be more appropriate as a studying ground. For example, if you are studying programming languages theory, it might be better to stick with Scheme or LISP so that you can more easily translate your Lambda calculus and tail recursions and continuations into working code... (and no, I don't want to reopen the "Lisp vs Perl" discussion :-) --ZZamboni	[reply]
Re^2: Benchmarking with Memory Profiling by shiza (Hermit) on Jun 14, 2005 at 18:31 UTC
valgrind (http://valgrind.org) may be of use.	[reply]
Re: Benchmarking with Memory Profiling by bikeNomad (Priest) on Jun 10, 2001 at 07:23 UTC
You could look at the output of perl -Dm (see perlrun). It allegedly gives memory allocation data. You have to compile Perl with debugging enabled first. Or you could just use the tools on your operating system for watching process memory usage. You can also look into Devel::DProf and Devel::SmallProf to get counts of the number of times subroutines or lines are getting executed.	[reply]
Re: Benchmarking with Memory Profiling by bikeNomad (Priest) on Jun 10, 2001 at 22:51 UTC
As far as learning algorithms using Perl, you're more likely to learn something if you can make test programs quickly to compare algorithms. And Perl is really good for that. Jon Bentley had a column from 1983 through 1985 in CACM called "Programming Pearls" in which he discussed algorithm design using AWK as a programming language. He was able to discuss algorithmic complexity and optimization even though AWK didn't have any built-in tools for doing this. He collected a number of these columns in his book Programming Pearls. Sadly, he re-wrote it using C and C++. It was easier to read in AWK, I think. Perl should have the tools you need for your exploration.	[reply]
Re: Benchmarking with Memory Profiling by John M. Dlugosz (Monsignor) on Jun 10, 2001 at 21:57 UTC
There is an O'Reilly book about algorithms with Perl, but I don't recall the name or animal (Update: It's a Wolf). I found that the study of data structures and algorithms, early on, was the most valuable class in my programming career. Tinkering with it was key to understanding things like how a binary tree keeps things "in order", so I agree that a language you can tinker in is a plus. —John	[reply]
Re: Re: Benchmarking with Memory Profiling by Beatnik (Parson) on Jun 10, 2001 at 22:32 UTC
Perl in a nutshell (Camel head on cover) isn't about algorithms, and Mastering Algorithms (Wolf on cover) with Perl isn't a Nutshell :) Greetz Beatnik ... Quidquid perl dictum sit, altum viditur.	[reply]
Re: Re: Benchmarking with Memory Profiling by sierrathedog04 (Hermit) on Jun 11, 2001 at 20:15 UTC
Mastering Algorithms with Perl by Jon Orwant and Jarkko Hietaniemi The animal on the cover seems to be some kind of dog, wolf or jackal but I cannot tell which. It might even be Dr. Orwant himself, who sometimes goes under the name "Mad Dog."	[reply]
Re: Benchmarking with Memory Profiling by mattr (Curate) on Jun 11, 2001 at 10:28 UTC
Ditto the comments on mod_perl documentation. You may want to check out Apache::VMonitor which lets you view an advanced scoreboard of system stats. You can edit this module to add your own stats too. One thing to be careful about, if you are looking at memory and run more than one mod_perl child, they will be sharing memory efficiently but may seem on the scoreboard to be taking up extra space (it does say "shared" for that value though) Requires GTop, Time::HiRes and Apache::Scoreboard.	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks

Benchmarking with Memory Profiling

Mastering Algorithms with Perl by Jon Orwant and Jarkko Hietaniemi