memory penalty for loading XS modules?

lestrrat has asked for the wisdom of the Perl Monks concerning the following question:

I'm currently trying to port DateTime::TimeZone to XS so that it consumes less memory. But I'm currently being baffled by the seeming *increase* in the memory being used by loading the XS version of the module.

DateTime::TimeZone is comprised of a a base class, DateTime::TimeZone, and a few hundred subclasses which represent each time zone. For example, Asia/Tokyo timezone is represented as DateTime::TimeZone::Asia::Tokyo. Each of these timezones contains a list of data, which I've now converted to C structs.

Now, loading one time zone and comparing its size with what Devel::Size reports, the savings are clear. Here's an example:

   Europe/Belfast: 51735 (Pure Perl)
   Europe/Belfast: 2525 (XS)
[download]

However, this isn't translating into actual memory savings. This is the output from my top (Mac OS X)

     PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSI
+ZE  VSIZE
(XS) 26873 perl         0.0%  0:03.43   1    12  1177  14.6M  12.0M  2
+1.9M  51.0M 
(PP) 26872 perl         0.0%  0:02.83   1    12    46  13.4M  1.42M  1
+4.6M  33.3M
[download]

What gives? The XS version seems to consume more than 18 MB more than the pure perl version?! I think I've applied all the XS hackery I know to reduce memory consumption, is there any tips to reduce this memory size? or am I looking at the wrong numbers?

Comment on memory penalty for loading XS modules? Select or Download Code

Replies are listed 'Best First'.
Re: memory penalty for loading XS modules? by salva (Canon) on Jul 11, 2005 at 16:37 UTC
without seeing your code, we can not really help you... anyway, Devel::Size doesn't take into account memory consummed by non-perl structures. Are you compiling the C module with debugging information, doing so can add a lot of "rubbish" to the library... though usually no so much! Check that you are passing the right size to the perl memory functions and macros, their argument types and order are inusual and passing the wrong arguments is not so rare!	[reply]
Re^2: memory penalty for loading XS modules? by lestrrat (Deacon) on Jul 12, 2005 at 13:09 UTC
Sheesh, part of it was the stupid -ggdb flag. Still there's a 10 MB penalty. `PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE + VSIZE 17233 perl 0.0% 0:03.03 1 12 46 13.2M 1.42M 14.4M + 33.3M 17232 perl 0.0% 0:05.40 1 12 1176 14.1M 7.79M 20.2M + 43.8M` [download] Still experimenting...	[reply] [d/l]
Re: memory penalty for loading XS modules? by hatter (Pilgrim) on Jul 11, 2005 at 16:41 UTC
Fast, small, correct. Pick two, if you're having a good day. Otherwise, be happy with one; some days you'll not get any. `the hatter`	[reply] [d/l]
Re^2: memory penalty for loading XS modules? by salva (Canon) on Jul 12, 2005 at 09:02 UTC
do you really think so? IMHO small code is usually also fast and correct, it is when you start adding (unneeded) features that it becomes big, slow and buggy!	[reply]
Release Early, Release Often. by mugwumpjism (Hermit) on Jul 12, 2005 at 06:08 UTC
Go ahead and release your module to CPAN with an underscore in the version number, so that it is marked as a development release not suitable for indexing. Then people can scratch their heads over it to figure out what's wrong ;-). $h=$ENV{HOME};my@q=split/\n\n/,`cat $h/.quotes`;$s="$h/." ."signature";$t=`cat $s`;print$t,"\n",$q[rand($#q)],"\n"; [download]	[reply] [d/l]
Re: Release Early, Release Often. by Anonymous Monk on Jul 12, 2005 at 06:43 UTC
Why not throw some more broken code on CPAN and hope someone else will fix it.	[reply]
Re^2: Release Early, Release Often. by mugwumpjism (Hermit) on Jul 14, 2005 at 04:18 UTC
Yes, you see CPAN is the Comprehensive Perl Archive Network. Not the Constrained Perl Archive Network. If you can't understand or deal with it and are forced to make snide comments about the state of something that is working as designed, then that is unfortunately your problem. $h=$ENV{HOME};my@q=split/\n\n/,`cat $h/.quotes`;$s="$h/." ."signature";$t=`cat $s`;print$t,"\n",$q[rand($#q)],"\n"; [download]	[reply] [d/l]
Re: memory penalty for loading XS modules? by sth (Priest) on Jul 12, 2005 at 18:55 UTC
lestrrat I would post this to the perl-xs mailing list. May be a couple of days before you get a reply, but it will be worth posting there.	[reply]