comment on

So my original code contained the following.

my $twig = 'XML::Twig'->new;
$twig->xparse($input_xml);
[download]

And POJ's code contains the following.

my $twig = 'XML::Twig'->new( twig_handlers =>  { result => \&record })
+;
$twig->parse($input_xml);
[download]

POJ's code when I do 10000 records is blowing up in the Twig module as follows.

  DB<2> c
Exiting subroutine via next at vance.pl line 91.
 at vance.pl line 91.
    main::record(XML::Twig=HASH(0x6003ead30), XML::Twig::Elt=HASH(0x60
+2c20e90)) called at /usr/lib/perl5/site_perl/5.22/XML/Twig.pm line 23
+48
    XML::Twig::_twig_end(XML::Parser::Expat=HASH(0x6003fafe0), "result
+") called at /usr/lib/perl5/site_perl/5.22/x86_64-cygwin-threads/XML/
+Parser/Expat.pm line 474
    XML::Parser::Expat::parse(XML::Parser::Expat=HASH(0x6003fafe0), "<
+?xml version=\"1.0\" encoding=\"UTF-8\"?><response><result><chec"...)
+ called at /usr/lib/perl5/site_perl/5.22/x86_64-cygwin-threads/XML/Pa
+rser.pm line 187
    eval {...} called at /usr/lib/perl5/site_perl/5.22/x86_64-cygwin-t
+hreads/XML/Parser.pm line 186
    XML::Parser::parse(XML::Twig=HASH(0x6003ead30), "<?xml version=\"1
+.0\" encoding=\"UTF-8\"?><response><result><chec"...) called at /usr/
+lib/perl5/site_perl/5.22/XML/Twig.pm line 767
    eval {...} called at /usr/lib/perl5/site_perl/5.22/XML/Twig.pm lin
+e 767
    XML::Twig::parse(XML::Twig=HASH(0x6003ead30), "<?xml version=\"1.0
+\" encoding=\"UTF-8\"?><response><result><chec"...) called at vance.p
+l line 33
Exiting subroutine via next at vance.pl line 91.
[download]

It is deep in the Twig module when I single step, but I think its tied to some sort of overflow condition. Any ideas as to how to fix this? POJ, why did you change those lines? Here is the current code.

use strict;
use warnings;
use XML::Twig;
use Text::CSV;
use Data::Dump 'pp';
use Data::Dumper;
use REST::Client;
use MIME::Base64;
use Text::Unidecode;
use utf8;

my @header   = ();
my @csv_rows = ();
my $order_by = 'u_ci_id';

my %seen = ();
my %arx_data = ();

# parse xml in chunks
my $twig = 'XML::Twig'->new( twig_handlers =>  { result => \&record })
+;
my $client = REST::Client->new(host => 'https://staplessb.service-now.
+com');
my $encoded_auth = encode_base64("1774580:Natalie1", '');

$client->GET("/api/now/table/cmdb_ci?sysparm_limit=10000",

                {'Authorization' => "Basic $encoded_auth",

                 'Accept' => 'application/xml'});

my $input_xml = $client->responseContent();

$twig->parse($input_xml);

# data dump
pp \%arx_data;
#pp \@csv_rows;
#pp \@header;

# sort and dump csv
my @sorted = sort { $b->[0] cmp $a->[0] } @csv_rows;
unshift @sorted,[@header]; # add header

my $outfile = 'xmldump.csv';
my $csv = Text::CSV->new ( { binary => 1 } )  
    or die "Cannot use CSV: ".Text::CSV->error_diag ();
$csv->eol("\n"); 

if ($outfile){        
  open my $fh, ">:encoding(utf8)", $outfile or die "$outfile: $!";
  for (@sorted){
    shift @$_; # remove sort field
    $csv->print ($fh, $_);
  }
  close $fh or die "new.csv: $!"; 
} else {
  for (@sorted){
    shift @$_;
    $csv->print (\*STDOUT, $_);
  }
}  

# parse 1 record
sub record {
  my ($e,$t) = @_;
  
  # csv records
  unless (@header){
    my @info_tags = $t->children;
    @header = map{ $_->name } @info_tags;
    unshift @header,$order_by; # add sort field
  }
  
  # extract data in same order as header row
  my @data = map{ $t->field($_) } @header;
  push @csv_rows,clean(@data); #returns array ref
 
  # build hash structure
  my $app   = $t->field('u_application_id');
  my $u_ci  = $t->field('u_ci_id');
  my $class = $t->field('sys_class_name');
  my $name  = $t->field('name');
  my $ip    = $t->field('ip_address');
  my $fqdn  = $t->field('fqdn');
  my $dns   = $t->field('dns_domain');
  my $mac   = $t->field('mac');
  
  if ( $class eq 'cmdb_ci_appl'){
    $arx_data{$u_ci}{'name'} = $name;  
  } else {
    next if $seen{$app}{$ip}++;
    push @{$arx_data{$app}{'members'}},$ip;
  }
  
}

sub clean {
  my @f = @_;
  for (@f){
    s/\r|\n//g;  # Cleanup Carraige Returns
    s/, / /g;    # Cleanup Comma Space
    s/,/ /g;     # Cleanup Comma
    s/"//g;      # Cleanup Parentheses
    s/^\s+|\s+$//g;    # Trim spaces
    s/([^[:ascii:]]+)/unidecode($1)/ge;
  }
  return \@f;
}
[download]

In reply to Re^4: Need help with complex hash of hashes. by vlturner
in thread Need help with complex hash of hashes. by vlturner

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.