What have you tried? Without looking at the XML::Twig documentation, I would set up a handler for the close/end of the outer most element that I cared about, at which point all the elements/attributes inside would be available. I'd use the info there, insert it to the db, flush it, and return from the callback. This should keep memory usage at a minimum, speed at a maximum, and keep things simple...ish.

Why say, that is something like http://web.archive.org/web/20070819113402/http://www.xmltwig.com/tutorial/perl_xml/mtb04_73.html, Re: Parse XML into CSV Files

I added the current code I am trying to use below however not positive I did it correctly. From what I was reading in the tutorial, I am grabbing the events where the <file> gets parsed, uploading it to the sql, then trashing the parse and moving to the next one. I am getting an error that I'm trying to get resolved atm but any feedback would be much appreciated.

<?xml version="1.0" encoding="utf-8"?>
<file Catid="0011" Date_added="20050910000000" HighPic="photo/location
+.jpg" HighPicHeight="123" HighPicSize="1122" HighPicWidth="123" Model
+="varchar model name" On_Market="0" Prod_ID="varchar id name" Product
+_I="111" Product_View="11223" Quality="qualitylevel" Supplier="x" Upd
+ated="20110713144032" path="required/path/to/other.doc"></file>
[download]

#!/usr/bin/perl

use strict;
use Data::Dumper;
use XML::Simple qw(:strict);

my $xs = XML::Simple->new();

my $xml = $xs->XMLin('/root/Desktop/914477.xml',
           ForceArray => 1,
           KeyAttr => 1);
binmode STDOUT, ':encoding(utf8)';            
print Dumper($xml);
[download]

Thanks for the code but this is what I have so far

# !/usr/local/bin/perl -w

BEGIN {
    my $base_module_dir = (-d '/home/bcnagle/perl' ? '/home/bcnagle/pe
+rl' : ( getpwuid($>) )[7] . '/perl/');
    unshift @INC, map { $base_module_dir . $_ } @INC;
}

use strict;
use XML::Twig;
use DBI;

my $path;
my $Product_ID;
my $Updated;
my $Quality;
my $Supplier_id;
my $Prod_ID;
my $Catid;
my $On_Market;
my $Model_Name;
my $Product_View;
my $HighPic;
my $HighPicSize;
my $HighPicWidth;
my $HighPicHeight;
my $Date_Added;

my $dbh= connect_to_db();
my $insert= $dbh->prepare(  "INSERT INTO files (path, Product_ID, Upda
+ted, Quality, Supplier_id, Prod_ID, Catid, On_Market, Model_Name, Pro
+duct_View, HighPic, HighPicSize, HighPicWidth, HighPicHeight, Date_Ad
+ded) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);");

my $twig = new XML::Twig(
    twig_handlers =>
    {indexfile => \$file}
    );

$twig->parsefile( "/file.xml" );
$twig->flush;                                     # flush the end of t
+he twig
$dbh->disconnect();
exit;

sub connect_to_db
  { my $driver = "mysql";
    my $dsn = "DBI:$driver:database=database;";
    my $dbh = DBI->connect($dsn, 'uname', 'pword', {AutoCommit=>1});
    my $drh = DBI->install_driver($driver);
    return( $dbh);
  }

sub indexfile
  { my($twig, $file) = @_;                        

    $path = $file->att( 'path');
    $Product_ID = $file->att( 'Product_ID');
    $Updated = $file->att( 'Updated');
    $Quality = $file->att( 'Quality');
    $Supplier_id = $file->att( 'Supplier_id');
    $Prod_ID = $file->att( 'Prod_ID');
    $Catid = $file->att( 'Catid');
    $On_Market = $file->att( 'On_Market');
    $Model_Name = $file->att( 'Model_Name');
    $Product_View = $file->att( 'Product_View');
    $HighPic = $file->att( 'HighPic');
    $HighPicSize = $file->att( 'HighPicSize');
    $HighPicWidth = $file->att( 'HighPicWidth');
    $HighPicHeight = $file->att( 'HighPicHeight');
    $Date_Added = $file->att( 'Date_Added');

    $insert->bind_param( 1, $path);
    $insert->bind_param( 2, $Product_ID);
    $insert->bind_param( 3, $Updated);
    $insert->bind_param( 4, $Quality);
    $insert->bind_param( 5, $Supplier_id);
    $insert->bind_param( 6, $Prod_ID);
    $insert->bind_param( 7, $Catid);
    $insert->bind_param( 8, $On_Market);
    $insert->bind_param( 9, $Model_Name);
    $insert->bind_param( 10, $Product_View);
    $insert->bind_param( 11, $HighPic);
    $insert->bind_param( 12, $HighPicSize);
    $insert->bind_param( 13, $HighPicWidth);
    $insert->bind_param( 14, $HighPicHeight);
    $insert->bind_param( 15, $Date_Added);

    $insert->execute();

    $twig->purge;
  
  }
[download]

I am getting a shell error when running the script stating that

weaken is only available with the XS version of Scalar::Util at /home/bcnagle/perl/usr/lib/perl5/site_perl/5.8.8/XML/Twig.pm line 117 BEGIN failed--compilation aborted at /home/bcnagle/perl/usr/lib/perl5/site_perl/5.8.8/XML/Twig.pm line 172. Compilation failed in require at /home/bcnagle/public_html/xmlparse.pl line 9. BEGIN failed--compilation aborted at /home/bcnagle/public_html/xmlparse.pl line 9.

You are probably using a RedHat-based distribution, which includes Scalar::Util without its XS part, and so doesn't provide weaken (see Task::Weaken). The solution would be to re-install Scalar::Util, from CPAN.

Also, I am not sure why you do a $twig->flush; at the end of your code, since you are not outputting the XML at all.

I also can't clean up the xml for it's a fixed xml from a manufacturer database for distribution. The file is extremely big so must parse line by line or else the server may cancel the script. looking into switching hosting company for many issues, this being one of them.

Tutorial, both perl, xml,... and XML::Twig, http://web.archive.org/web/20090306200947/http://xmltwig.com/tutorial/perl_xml/

site:perlmonks.org XML::Twig csv

xml to csv

Parse XML into CSV Files

Using XML::XSLT to convert XML to CSV

PerlMonks FAQ: How do I post a question effectively?, How do I compose an effective node title?, Is PM a good place to get answers for homework?

so $insert->bind_param( 1, $path, 2, etc, 3, etc, 4 etc, 5, etc, etc, etc); ??

Um, I said move, not rewrite :) Move as in

my $insert = ...
...
sub indexfile {
  ...
  $insert->bind_param( ...
  ...
  $insert->execute();
[download]

becomes

my $insert = ...
  $insert->bind_param( ...
  ...
...

sub indexfile {
   ...
   $insert->execute();
[download]

you prepare once, you bind once, you execute many many many many times, like each time you update your bound params :)