Dynamically breaking up a data set

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Monks, I have code which generates a large XML packet (containing hundreds of packet_refs, see below) based on the result of a SQL query. I want to break this down so that 1 XML packet, each containing 10 packet_ref's is sent, then looped for the rest of the results. I've tried to adapt my code to do this, but I don't know what the condition should be for the if statement. As requested, this is a cut down example of my code. Please advise

foreach my $item (@SQLvars){
    
    my $sth = $self->dbh()->prepare($DocumentQuery) 
        or die "Can't prepair statement: $self->DBI::errstr";
    
    my ( $foo, $bar );
    
    $sth->execute( 
        $item->{PACKID}, 
        $item->{PACKINFO}
        );
    
    $sth->bind_columns(\( $foo, $bar ));
    
    while ($sth->fetch) {

        $XMLContent .= "<packet_ref>
                        <packID>$foo</packID>
                        <packInfo>$bar</packINFO>
                    </packet_ref>\n";
                    
        if (){
            # if there are 10 packet_ref's send the XML and loop round
+ for the remainder
            my $XMLPacket;
            $XMLPacket = $XMLHead . $XMLContent . $XMLFoot;
            
            my $mech = WWW::Mechanize->new();

            $mech->get( $TargetURL );
            $mech->field( "xml_doc", $XMLPacket );
            my $mechres = $mech->submit();

            if ($mechres->is_success) {
                warn $mechres->decoded_content;
                return "Success";
            }else{
                warn $mechres->status_line, "\n";
                return "Error";
            }
            undef $XMLContent;
        }            


    }
        
}
[download]

Comment on Dynamically breaking up a data set Download Code

Replies are listed 'Best First'.
Re: Dynamically breaking up a data set by choroba (Cardinal) on May 23, 2012 at 12:02 UTC
Why can't you just count the packets in $packet_count, and if there are 10 of them, just start counting from 0 again?	[reply]
Re^2: Dynamically breaking up a data set by Anonymous Monk on May 23, 2012 at 12:22 UTC
I'm new to all this, could you please give me a hit where to initialize, ++ and undef this variable?	[reply]
Re^3: Dynamically breaking up a data set by choroba (Cardinal) on May 23, 2012 at 12:32 UTC
Declare it before the loop so its value is kept on each iteration. ++ it on each packet, clear it when you output the XML (i.e. after 10 steps).	[reply]
Re^4: Dynamically breaking up a data set by Anonymous Monk on May 23, 2012 at 14:34 UTC
Re: Dynamically breaking up a data set by Anonymous Monk on May 23, 2012 at 12:02 UTC
For FMS's sake, use an XML library! You're creating invalid syntax.	[reply]
Re: Dynamically breaking up a data set by locked_user sundialsvc4 (Abbot) on May 23, 2012 at 12:47 UTC
I strongly suggest that you use an XML-processing CPAN package such as, say, XML::LibXML, and that you “query” the file using an XPath expression. If you select and use a module that is designed to work with large XML files, will provide you with the means of obtaining and iterating through a list of nodes that are extracted from the file, no matter how big the file may be. By using XPath expressions, you specify what you are interested in and do not have to write one-off Perl logic to navigate through the structure. You might also be able to get even further along with your task by using so-called XSLT stylesheets (which are not at all like CSS stylesheets). Very sophisticated XML transformations and queries can be specified using no programming per se at all. You might even find that you can do the entire task this way, using command-line tools and actually eliminating Perl altogether.
Re^2: Dynamically breaking up a data set by Jenda (Abbot) on May 24, 2012 at 08:17 UTC
Erm .... sorry to bother you with details, but ... the OP is not querying any XML. He's not parsing any XML file and while I understand that someone twisted enough might like XSLT for XML-to-something transformation, in this case that's not gonna fly. There is no XML to start with! The data are in a database and the XML is to be produced. The data should be escaped though to prevent problems if the PACKINFO contains less-than signs or other stuff special to XML. Eg. like this: `use XML::Rules; my $parser=new XML::Rules(rules=>{}); ... $XMLContent .= $parser->toXML(packet_ref => {packID => [$foo], packInf +o => [$bar]}) . "\n"; # or, with formatting # $XMLContent .= $parser->toXML(packet_ref => {packID => [$foo], packI +nfo => [$bar]}, 0, ' ') . "\n";` [download] Jenda Enoch was right! Enjoy the last years of Rome.	[reply] [d/l]