You're running out of memory because the file contains so much data. The data structure your XML module builds is too big for your machine. The solution is to use a XML parser that can handle stream mode and doesn't attempt to build the whole document.

XML::Twig would be a good one to try. Its pod has an example of working with a huge file.

Can you give me an example of the code above in xml::twig?

XML::Parser is a good candidate here.

All you want is to go through the XML document once, and process each group of tags one by one. At any given time, you only need a handful of those tags and it content stored in memory.

Put your code database access code in handlers.

With XML::Parser, the parser only does the parsing (which is what you need), but the rest is up to you, including whether you want to store the XML file in a data structure, or how big a portion of it should be stored in the memory at a given time. As I said, in your case, all you want at a given time is the content of a handful tags.

While the other posts have some insight into some potential problems, I'll add my most recent out-of-memory error solutions:

• ulimit - make sure your memory settings are set as high as you need. If you have a box unto yourself, the ulimit could be set to unlimited - if it is already set there, then this isn't your problem.
• Perl version. You don't specify what perl version you're using, but we had to undergo an emergency upgrade from 5.8.6 to 5.8.7 which solved some of our out-of-memory conditions. Also, when I first started using XML, we were using 5.6.0 even though 5.8.0 was out. Under 5.6, we got constant memory problems, while 5.8.1 (which we eventually moved to for that prototype) had none, with no other changes to the code.

• Instead of $db->do("begin"); the 'AutoCommit' DBI attribute is typically used:

  my $db = DBI->connect($data_source, $username, $auth, { AutoCommit => 
  +0 } );
  [download]

• might as well placeholder the DELETE statement, too (and don't need to prepare):

  $del = $db->do(delete from master where org = ?", {}, $we);
  [download]

• You can use a slice of the hashref to make the bind variables easier to read/maintain:

  $db2->execute(
    $we,
    $id,
    @{ $response->{offer}->{$id} }{ qw/
      city
      state
      zip
      link
      slots_available
      description
    / },
  );
  [download]

For those living off-planet the last couple of weeks: Hurricane Katrina. ;-)

Super Search is everyone's friend on PerlMonks. These nodes may be of help to you:

Memory errors while processing 2GB XML file with XML:Twig on Windows 2000

splitting large xml documents

Don't forget to take a look at the docs for XML::Twig and XML::Parser.

If you come up with more code but are still having a problem, let us know and I'm sure someMonk will be happy to help.

HTH,