Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

First of all, I'm not a genius when it comes to this stuff, so please be kind.

Platform is ActiveState 5.6.1 Build 633 on WinME (stop laughing).
DBI is version 0.2002
Win32::GUI (Which as I understand it contains OLE) version 0.0.558
I last verify --upgraded about two weeks ago. I tried to do it again when I encountered this problem, but ppm seems to be a little fussy right now.

A little background: I work for a bookkeeping company that uses QuickBooks (2002, we haven't yet upgraded to 2003, and upgrading probably won't fix this problem.). As of QB 2002 (US Version), Intuit has integrated something they call the SDK, which allows other programs to at least "talk" to QuickBooks in XML. So far, I like it. I've built some very useful programs using it.

The catalyst: Until recently, we've been using the QuickBooks Timer. We stopped using it simply because Intuit hasn't bothered to updated it since 1998 and it's starting to cause serious problems (system lockups and lost time), so we switched to another timer. Unfortunately, this timer doesn't integrate with QuickBooks very well - as in not at all. But the new timer does allow us to export to CSV.

So, I built a program to grab the stuff out of the CSV and put it in QB. And it should have worked. Seriously. The XML being sent to QuickBooks was perfectly valid. As a matter of fact, I ran it QB's own validator and submitted through the XML Feeder (a small utility I made to simply "feed" the xml to QuickBooks to make sure the XML will do what I want it to) and everything went through without so much as a sniffle.

Enough chit-chat. I think I've narrowed down the problem to some kind of conflict between DBI and Win32::OLE.

The following program is something I whipped together, basically, it takes DBI out of the loop. I know you shouldn't hand-roll a CSV parser, but I had to (only with controlled sample data) in order to narrow down the problem. The final program WILL NOT reinvent any wheel that I know exists. The final program also handles errors better and also checks the return XML from QB for problems. And the commenting is also better in the final program. This was just a test.

#!perl -Tw use strict; use Win32::OLE; use XML::Generator; my $list_id = '40000-1022206382'; my $request_id = 1; # Be paranoid about error messages Win32::OLE->Option(Warn => 3); # Start the Processor my $connection = Win32::OLE->new('QBXMLRP.RequestProcessor'); # Name your program $connection->OpenConnection("qbtmr", "Timer Import"); # Open the currently open QuickBooks file, in "DoNotCare" mode. my $ticket = $connection->BeginSession('', 2); my @data = <DATA>; foreach my $line (@data) { my ($customer, $service, $duration, $date) = split(/,/, $line); my $gen = XML::Generator->new( escape => 'always', conformance => 'strict', pretty => 2, dtd => [ 'QBXML', 'PUBLIC', "'-//INTUIT//DTD", " +QBXML QBD 1.1//EN'"], version => '1.0' ); my $xml = $gen->xmldecl; $xml = $gen->QBXML( $gen->QBXMLMsgsRq( {'onError' => 'stopOnError'}, $gen->TimeTrackingAddRq({requestID => 1}, $gen->TimeTrackingAdd( $gen->TxnDate($date), $gen->EntityRef( $gen->ListID($list_id) ), $gen->CustomerRef( $gen->FullName($customer) ), $gen->ItemServiceRef( $gen->FullName($service) ), $gen->Duration($duration), ), ), ), ); my $xml_req = $gen->xml($xml); $request_id++; open (REINVENT, ">reinvent.xml") or die "Can't open reinvent.xml f +or writing - $!"; print REINVENT $xml_req; close REINVENT; my $xml_res= $connection->ProcessRequest($ticket, $xml_req); } # Gracefully end everything $connection->EndSession($ticket); $connection->CloseConnection; __DATA__ Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AR,PT00H01M,2003-01-01

The next program uses DBI to parse the CSV. It's virtually identical, with the exception of the DBI modifications.

#!perl -Tw use strict; use DBI; use Win32::OLE; use XML::Generator; my $list_id = '40000-1022206382'; my $request_id = 1; # Be paranoid about error messages Win32::OLE->Option(Warn => 3); # Start the Processor my $connection = Win32::OLE->new('QBXMLRP.RequestProcessor'); # Name your program $connection->OpenConnection("qbtmr", "Timer Import"); # Open the currently open QuickBooks file, in "DoNotCare" mode. my $ticket = $connection->BeginSession('', 2); my $dbh = DBI->connect("DBI:CSV:") or die "Cannot connect: " . $DBI::errstr; my $sth = $dbh->prepare("SELECT Customer, Service, Duration, Date FROM + data") or die "Cannot prepare: " . $dbh->errstr(); $sth->execute() or die "Cannot execute: " . $sth->errstr(); $sth->bind_columns(\my($customer, $service, $duration, $date)); while ($sth->fetch) { my $gen = XML::Generator->new( escape => 'always', conformance => 'strict', pretty => 2, dtd => [ 'QBXML', 'PUBLIC', "'-//INTUIT//DTD", " +QBXML QBD 1.1//EN'"], version => '1.0' ); my $xml = $gen->xmldecl; $xml = $gen->QBXML( $gen->QBXMLMsgsRq( {'onError' => 'stopOnError'}, $gen->TimeTrackingAddRq({requestID => 1}, $gen->TimeTrackingAdd( $gen->TxnDate($date), $gen->EntityRef( $gen->ListID($list_id) ), $gen->CustomerRef( $gen->FullName($customer) ), $gen->ItemServiceRef( $gen->FullName($service) ), $gen->Duration($duration), ), ), ), ); my $xml_req = $gen->xml($xml); $request_id++; open (ACTUAL, ">actual.xml") or die "Can't open actual.xml for wri +ting - $!"; print ACTUAL $xml_req; close ACTUAL; my $xml_res= $connection->ProcessRequest($ticket, $xml_req); } $sth->finish(); $dbh->disconnect(); # Gracefully end everything $connection->EndSession($ticket); $connection->CloseConnection;

And the data file that goes with the DBI one. Also virtually identical to the data in the first program, with the exception of column names:

Customer,Service,Duration,Date Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AR,PT00H01M,2003-01-01

Now, the first program goes off without a hitch. All the data is inserted into QuickBook's time tracking section.

The second program, on the other hand, pukes with this (line number omitted because it's not correct anyway, but the error is right on the money):

OLE exception from "QBXMLRP.RequestProcessor.1": QuickBooks found an error when parsing the provided XML text stream. Win32::OLE(0.1502) error 0x80040400 in METHOD/PROPERTYGET "ProcessRequest" at parse.pl line ###

And upon looking in the qbsdklog.txt in the QB directory, the error message is (Ignore lines 2-4 and 12-13, it's garbage):

20030102.134136 I -75547941 RequestProcessor ========= Sta +rted Connection ========= 20030102.134136 I -75547941 RequestProcessor Connection op +ened by app named 'Timer Import' 20030102.134136 E -75547941 CRLInstaller Failed to query r +egistry value. The parameter is incorrect. 20030102.134136 I -75547941 CertVerifier The file does not + contain an Authenticode signature. 20030102.134136 I -75547941 RequestProcessor Opening the f +ile in the DoNotCare mode. 20030102.134137 I -898753 QBSDKProcessRequest Application +named 'Timer Import' starting requests (process -75547941). 20030102.134137 E -898753 XMLDtdMapper Invalid DOCTYPE 20030102.134137 E -898753 XercesSAXParseManager Invalid DO +CTYPE - unable to transform from PUBLIC to SYSTEM 20030102.134137 I -898753 MsgSetHandler Parser error, fini +shing. 20030102.134137 I -898753 QBSDKProcessRequest Application +named 'Timer Import' finishing requests (process -75547941), ret = 1. 20030102.134137 E -75547941 RequestProcessor Process Reque +st error = 80042500, There was an error when parsing the provided XML + file. 20030102.134137 I -75547941 RequestProcessor "EndSession" +has not been called. 20030102.134137 I -75547941 RequestProcessor "CloseConnect +ion" has not been called.

The important thing to note is that both the XML from the first program and the XML from the second program are identical. As a matter of fact, if I use my "XML Feeder" to feed actual.xml (the XML from the second program - the "supposed" culprit) to QB, QB happily accepts it.

I tried looking in DBI and Win32::OLE for any conflict, but my eyes glaze over. I'm just not good enough at this to figure it out.

I'm at a loss. I really need this program as entering the CSV by hand takes almost 2 hours. I'd really find it surprising if DBI and Win32::OLE aren't getting along, but that's the only thing I can think of at this point.

So, I guess my question is, after looking at the code and data samples above, what am I doing wrong that makes the second program fail?

Sarah

edited: Fri Jan 3 04:21:07 2003 by jeffa - adjusted <readmore> tags

Replies are listed 'Best First'.
Re: DBI and Win32::OLE Conflict?
by diotalevi (Canon) on Jan 02, 2003 at 20:24 UTC

    How about using a signature to verify that both XML strings are the same? Also consider running something this against your data to look for otherwise invisible differences. Be sure to leave white space out of the regex - you want to see which values are being used.

    s/([^\w<>])/sprintf "\\%03o", ord $1/e

    Update I forgot to terminate the s/// expression and didn't add a /e which completely foiled the point of the exercise.


    Fun Fun Fun in the Fluffy Chair

      I apologize but I couldn't figure out how to get your snipit to work (it kept putting sprintf "\\%03o", ord $1 between the first and second characters of both xml files. I don't think I used it correctly.

      But I did a quick search on this site for "invisible differences" and was pointed to a thread that recommended something called Beyond Compare to compare the two files. I downloaded the program and used it and it says the two xml files are identical.

      How would I go about finding out the signatures? (I said I wasn't very good...nobody believed me :) )

Re: DBI and Win32::OLE Conflict?
by pfaut (Priest) on Jan 03, 2003 at 16:03 UTC

    After looking through all the replies here and seeing what you've tried so far, the only things I can think of are (a) something getting screwed up through an @EXPORT or (b) different DLL's getting mapped depending on the order of processing. Would it be possible for you to dump %INC at the end of the program run? It might be telling to trace through that and see what modules are included, then figure out which have DLL's. The answer might lie in the dependancy chains. Is there any way to trace module/DLL activation?

    --- print map { my ($m)=1<<hex($_)&11?' ':''; $m.=substr('AHJPacehklnorstu',hex($_),1) } split //,'2fde0abe76c36c914586c';

      Thank you for your help - I really do appreciate you wasting your time on this. Unfortunately, after taking Friday night and all day Saturday to work on this, I am unable to figure out what's wrong, much less solve the problem. Maybe in a few years I'll be smart enough, but for now I'll just give up and do by hand what the program should have been able to do much faster.

Re: DBI and Win32::OLE Conflict?
by Mr. Muskrat (Canon) on Jan 02, 2003 at 20:09 UTC

      Thank you. I will look into using that module instead. I'm not sure it will do what I need it to, but I will play with it first before making a determination.

      Unfortunately, I don't think the module will work. I need the output to look something like:

      <?xml version="1.0" standalone="no"?> <!DOCTYPE QBXML PUBLIC '-//INTUIT//DTD QBXML QBD 1.1//EN'> <QBXML> <QBXMLMsgsRq onError="stopOnError"> <TimeTrackingAddRq requestID="1"> <TimeTrackingAdd> <TxnDate>2003-01-01</TxnDate> <EntityRef> <ListID>40000-1022206382</ListID> </EntityRef> <CustomerRef> <FullName>Cahoots Sports Bar &amp; Grill</FullName> </CustomerRef> <ItemServiceRef> <FullName>AR</FullName> </ItemServiceRef> <Duration>PT00H01M</Duration> </TimeTrackingAdd> </TimeTrackingAddRq> </QBXMLMsgsRq> </QBXML>

      For each line, a separate XML needs to be sent to QB because QB is a little quirky with their errors, even if I change "stopOnError" to "continueOnError". I also need to do things with the data before it goes to XML (the Duration and Date are formatted from something else to what you see in the data)

      The module you recommended produces something like:

      <records> <record> <Customer>Cahoots Sports Bar &amp; Grill</Customer> <Service>AP</Service> <Duration>PT00H01M</Duration> <Date>2003-01-01</Date> </record> <record> <Customer>Cahoots Sports Bar &amp; Grill</Customer> <Service>AP</Service> <Duration>PT00H01M</Duration> <Date>2003-01-01</Date> </record> <record> <Customer>Cahoots Sports Bar &amp; Grill</Customer> <Service>AR</Service> <Duration>PT00H01M</Duration> <Date>2003-01-01</Date> </record> </records>

      And I'm a little confused by the documentation. But I will keep trying with this module anyway. I'm sure I'm missing some kind of customization point or hack to make it work the way I need it to.

Re: DBI and Win32::OLE Conflict?
by Anonymous Monk on Jan 02, 2003 at 20:12 UTC

    I really screwed up on those readmores. :( Would an editor or someone mind fixing it? Up until the code, it looks okay, but I messed up and somehow the all that code ended up being shown. And the last few paragraphs, which I wanted shown were omitted.

Re: DBI and Win32::OLE Conflict?
by poj (Abbot) on Jan 02, 2003 at 20:33 UTC
    A really wild guess..
    try changing the '&' to 'and' in your test data, this character can cause problems in XML
    poj

      I need to keep that ampersand in there. If I change it to 'and' the item will not go into QuickBooks because the customer name I would be supplying is different from the one in QuickBooks.

      Besides, this isn't the case, because the ampersand comes out as &amp; in the XML and QB slurps that up (as it always has) just fine.

      I really do appreciate your effort, though

        Ok sorry, I saw the previous reply too late
        I'm looking at the difference between the first and second cases. In the first case this code
        my @data = <DATA>; foreach my $line (@data) { my ($customer, $service, $duration, $date) = split(/,/, $line); print "$customer:$service:$duration:$date:\n"; } __DATA__ Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AP,PT00H01M,2003-01-01 Cahoots Sports Bar & Grill,AR,PT00H01M,2003-01-01

        shows the $date field has a \n on it.
        Not sure if this is significant but worth a look
        poj