I strongly suggest joining the Mail::Box mailing list (see perl.overmeer.net) and posting your issue to the list -- the module author, Mark Overmeer, and many knowledgeable users are pretty quick to respond.

"30000" smells like "signed 16-bit overflow". This problem may even be in an underlying system library (because Mail::Box doesn't use XS).

I'd start by investigating which mail box format the production environment uses.

Production uses maildir only.

Sorry, fglock, but I can't see the point. Why this smells like an overflow? File system is out of question, 30_000 files is large but not really a problem...

-- monsieur_champs

There are some numbers in computing, which are boundries, and can cause problems (sort of like that whole y2k issue).

Near 30,000 is the number 32,768, which is 2**15. Now, you'd think to yourself, but wouldn't there be problems at 2**16, which is a nice round number in computer terms?

Well, no, because for an integer of (x) bits, if it's signed, it ranges from (-1*(2**(x-1)-1) to 2**(x-1). So, for a 16 bit number, it goes from -32767 to 32768. If the module in question uses XS (compiled C code), it's possible that it was compiled with a 16 bit signed number in there, which will have problems if you try to deal with numbers greater than 32,768.

If the number is exactly 30,000 or less, this probably isn't the issue. If it's over 32,768, this could be a problem.

From looking at the docs for Mail::Box, however, it looks to be pure perl, so I don't think this is the issue in this case.

still, it could be reasonable to write simple test script to open that dir and just count:

opendir DIR, "that-dir";
my @files = readdir DIR; # or whatever methos Mail::Box uses
print "count is ", $#files+1;
[download]

Just to check if perl's readdir is out of question on your current build (with given libc and so on)

Well, you know what happens when you presume, right? You make a pre of su and me.

This would be classified as the "easy" way out. I know programmers love to do things the hard way.

That's a nice workaround. Solving the problem the hard way will probably help others not to be biten in the future.

Flavio
perl -ple'$_=reverse' <<<ti.xittelop@oivalf

Don't fool yourself.

An off-the-wall guess that I have no way to verify. Could it be that you are running out of memory? A quick browse of Mail::Box and it's associated modules leads me to believe that they form a quite heavily nested hierarchy of modules with each level adding another hash or two for each item. Large volumes of nested hashes, even when each individual leaf hash is quite small, can rapidly consume large volumes of space.

On my system, Perl sometimes dies silently when it runs out of space.

Maybe you could monitor programs memory usage when running it on this large directory?

as of my personal experience, 5.6.1 version was unable to get more than 1Gb of memory, whereas 5.8.x versions behaved betted with respect to this (but was much slower on getting memory, BTW), so I would agree with your reasonable point.

Good point. But my test script stills printing all planned output, even that below the "count messages point". I suppose that an out-of-memory Perl shouldn't be able to print anything...

I'm awaiting for the test script results, so I can tell you more details. Please, if you think that the posted script is not enought to expose the problem, tell me, and I will try to write a more precise test. Patches welcome, too.

-- monsieur_champs

Hmm. Probably not memory then.

I took a quick scan of the code in Mail::Box::Manager and notice something that might be relevant. I the code for M::B::M::open(), I see this:

  return if $require_failed{$class};
[download]

and scanning back to see where $require_failed is being set and see this:

    unless($folder_type)
    {   # Try to autodetect foldertype.
        foreach (@{$self->{MBM_folder_types}})
        {   next unless $_;
            (my $abbrev, $class, @defaults) = @$_;
            next if $require_failed{$class};

            eval "require $class";
            if($@)
            {   $require_failed{$class}++;
                next;
            }

            if($class->foundIn($name, @defaults, %args))
            {   $folder_type = $abbrev;
                last;
            }
        }
     }
[download]

I may be misinterpreting the code, but it looks to me that if it attempts to auto detect the folder type and then fails to require the module for the folder type it detects, it sets the flag to indicate the failure and skips on without logging an error. Then later, it checks the flag and if it is set, fails silently returning undef.

Could it be misdetecting the folder type and failing silently as a result?

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.