powerhouse has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to create a form that I can use remotely that will enable me to do what the company that owns the site won't do.... Allow me to post all our invoices in one batch, instead of one at a time...

So I created a login where I can enter all the invoices in a text area.

Example in Textarea Post:
123,02928,06/28/2004,9.50
124,02893,06/28/2004,20.00

That is in this order....
Invoice Number, Store Number, Date, Amount

Then I have this code to do the processing for me...

        my $_invoices = param("invoices"); # Get the value of the text
+area
        if ($_invoices && $_invoices =~ /\015\012/) {
            $_split_val = "\015\012";
        } elsif ($_invoices && $_invoices =~ /\r\n/) {
            $_split_val = "\r\n";
        }

        use HTTP::Cookies;
        $cookie_jar = HTTP::Cookies->new(
            'file' => "/home/first/cook/lwp_cookies.dat",
            'autosave' => 1,
        );

        my $_post_url = 'http://www.eb2b.net/E2B_UMLogOn.process';
        my $ua = LWP::UserAgent->new;
        $ua->cookie_jar($cookie_jar);
  
        my $req = HTTP::Request->new(POST => "$_post_url");
        $req->content_type('application/x-www-form-urlencoded');
        $req->content("IWPEProcessFlow.submitted.sequenceID=&logonUser
+name=myhiddenusername&logonPassword=myhidden_password&Logon=Login&hMs
+gUserNotFound=User with this name was not found.&hMsgInvalidPassword=
+Logon not valid.&fromHomePage=default_B2B.htm");
        $req->referrer("http://www.eb2b.net/"); # Spoof their website.
+..
        my $res = $ua->request($req);
        if (!$res->is_success) {
            $_message = get_error_table("I could not login to the eb2b
+.net website.... please check what is up with that...");
            $in{view} = "start_form";
            batch_features();
            return;
        }
        my $_invoices_started = 0;
        my $_invoices_ended = 0;
        while($_invoices ne "") {
            $_invoices_started++;
            (my $_line_val, $_invoices) = split /$_split_val/, $_invoi
+ces, 2; # Split $_invoices and shorten by $_line_val....
            next if !$_line_val; # Does not contain anything go to the
+ next line....
            ($_inv_num,$_store_number,$_inv_date,$_amount) = split /\,
+/, $_line_val, 4;
                # Reset Page....
            $ua = LWP::UserAgent->new;
            $ua->cookie_jar($cookie_jar);
            $response = $ua->get('https://www.eb2b.net/E2B_OMInvoiceCr
+editMgmt.process?RestartFlow=t');

                # Start Template.... 
            $_post_url = 'http://www.eb2b.net/E2B_SelectTemplate.proce
+ss?restartFlow=t';
            $ua = LWP::UserAgent->new;
            $ua->cookie_jar($cookie_jar);
            $req = HTTP::Request->new(POST => "$_post_url");
            $req->content_type('application/x-www-form-urlencoded');
            $req->content("SelectTemplate=Use Template >");
            $req->referrer("https://www.eb2b.net/E2B_OMInvoiceCreditMg
+mt.process?RestartFlow=t");
            $res = $ua->request($req);
            
                # Create Invoice from Template...
            $_post_url = 'http://www.eb2b.net/E2B_OMInvoiceCreditMgmt.
+process?restartFlow=t&FromTemplate=t&Template_Id=522';
            $ua = LWP::UserAgent->new;
            $ua->cookie_jar($cookie_jar);
            $req = HTTP::Request->new(POST => "$_post_url");
            $req->content_type('application/x-www-form-urlencoded');
            $req->content("Relationship_Index=0&CInvoice=Create Invoic
+e");
            $req->referrer("https://www.eb2b.net/E2B_SelectTemplate.pr
+ocess?restartFlow=t");
            $res = $ua->request($req);

                # Post Invoice...
            $_post_url = 'http://www.eb2b.net/E2B_OMInvoiceCreditMgmt.
+process';
            $ua = LWP::UserAgent->new;
            $ua->cookie_jar($cookie_jar);
                # Create content into form....
            $_content = "E2B_OMInvoiceCreditForm.Purchase_Order_Number
+=&Location_Number=" . $_store_number . "&VendorRelationshipIndex=0&E2
+B_OMInvoiceCreditForm.Invoice_Number=" . $_inv_num . "&Purchase_Order
+_Date=&E2B_OMInvoiceCreditForm.Bill_Of_Lading_Number=&Invoice_Date=" 
+. $_inv_date . "&E2B_OMInvoiceCreditForm.Trailer_Number=&Ship_Date=" 
+. $_inv_date . "&E2B_OMInvoiceCreditForm.Fob_Code=CC&Total_Invoice_Ca
+rtons=1&Carrier_Code=ZZZZ&E2B_OMInvoiceCreditForm.Sales_Division_Numb
+er=6131195061&E2B_OMInvoiceCreditForm.Net_Invoice_Total=0&E2B_OMInvoi
+ceCreditForm.EditPageMarker=&E2B_OMInvoiceCreditForm.LineItemVector[0
+].Line_Number=1&E2B_OMInvoiceCreditForm.LineItemVector[0].Vendor_Item
+_Number=DD&E2B_OMInvoiceCreditForm.LineItemVector[0].Customer_Item_Nu
+mber=&E2B_OMInvoiceCreditForm.LineItemVector[0].Upc_Code=&E2B_OMInvoi
+ceCreditForm.LineItemVector[0].UPC_NDC_Indicator= &E2B_OMInvoiceCredi
+tForm.LineItemVector[0].Item_Description=Maintenance&E2B_OMInvoiceCre
+ditForm.LineItemVector[0].Quantity_Shipped=1&E2B_OMInvoiceCreditForm.
+LineItemVector[0].Unit_Of_Measure=EA&E2B_OMInvoiceCreditForm.LineItem
+Vector[0].Pack_Size=1&E2B_OMInvoiceCreditForm.LineItemVector[0].Unit_
+Price=" . $_amount . "&rcDirection=&selectedLine=&IWPEProcessFlow.sub
+mitted.sequenceID=/E2B_OMInvoiceCreditMgmt.process|runProcessE2B_OMMa
+nualInvoiceCredit|expandE2B_OMManualInvoiceCredit&Invoice_Complete_Fl
+ag=0&E2B_OMInvoiceCreditTermsAllowancesDirection=";
            $req = HTTP::Request->new(POST => "$_post_url");
            $req->content_type('application/x-www-form-urlencoded');
            $req->content($_content);
            $req->referrer("https://www.eb2b.net/E2B_OMInvoiceCreditMg
+mt.process?restartFlow=t&FromTemplate=t&Template_Id=522");
            $res = $ua->request($req);

                # Ok, now take the invoice just created and add the "a
+ccounting code" then send it to ekerdx
            $_post_url = 'http://www.eb2b.net/E2B_OMInvoiceCreditMgmt.
+process';
            $ua = LWP::UserAgent->new;
            $ua->cookie_jar($cookie_jar);
                # Create content into form....
            $_content = "E2B_OMInvoiceCreditTermsAllowancesDirection=&
+selectedAllowanceCollection=&selectedLine=&IWPEProcessFlow.submitted.
+sequenceID=/E2B_OMInvoiceCreditMgmt.process|runProcessE2B_OMInvoiceCr
+editTermsAllowances|expandE2B_OMInvoiceCreditTermsAllowances&Bus_Unit
+=00&Acct=664600&Cost_Dep_No=0000&RowSkip=&RowChunk=&TabChoice=&search
+By=&E2B_OMInvoiceCreditTermsAllowancesDirection=Confirm";
            $req = HTTP::Request->new(POST => "$_post_url");
            $req->content_type('application/x-www-form-urlencoded');
            $req->content($_content);
            $req->referrer("https://www.eb2b.net/E2B_OMInvoiceCreditMg
+mt.process");
            $res = $ua->request($req);
                # All Done....
            $_invoices_ended++;
        }
        $_page_content .= qq~Ok, I have started $_invoices_started inv
+oices and finished $_invoices_ended invoices!<br>
        <br>~;

[download]
This is not working, it is erroring out at my "cannot login" exit point.

Can someone please help me find out why this is not working?

Thank you!
Richard

Replies are listed 'Best First'.
Re: Remote Login and Form Posting
by Corion (Patriarch) on Jun 30, 2004 at 07:59 UTC

    Whenever you are automating a website, it is of utmost importance to trace what your "real" browser sends and then to try to replicate that with your script. I can recommend the Mozilla HTTP Liveheaders or HTTP::Recorder to sniff the traffic between your browser and the website.

    WWW::Mechanize specifically caters towards behaving like a "normal" browser, so you are maybe better off using WWW::Mechanize instead of LWP::UserAgent directly. All methods of LWP::UserAgent are available via WWW::Mechanize as well, so you don't lose anything, but you get magic redirect and referrer handling.

    While you are developing your website automation, you will often want to compare what your browser sends and what your script sends, and make the messages as identical as possible, so having a diff utility handy and pumping up the LWP logging might be helpfull too.

[reply]
      I have looked at that module and I don't see how to use a cookie_jar with it to first login to their website. I am looking through the module on cpan with the doc therein contained.

      I have put a TON of time into this and don't want to start at ground zero, unless I have to. Therefore, how would I get what I have already written to work?

      I added all the form fields to the LWP version I made, but it does not seem to be doing it right. Is there a way to make this code here work:
      use HTTP::Request;
use HTTP::Cookies;
my $_cookie_file = "/home/first/cook/.lwp_cookies.dat";
  $_post_url = 'http://www.eb2b.net/E2B_UMLogOn.process';
# Create new LWP::UserAgent object
  $browser = LWP::UserAgent->new;
# Add a new Cookie Jar to the LWP Object
  $browser->cookie_jar(HTTP::Cookies->new("file" => $_cookie_file));
# Spoof a very capable browser
  $browser->agent('Mozilla/5.0');
# Allow https protocol and http....
  $browser->protocols_allowed( [ 'http', 'https'] );
# Add the HTTP::Request for POST...
  $req = HTTP::Request->new("POST" => "$_post_url");
# Add "content type" as form encoded
  $req->content_type("application/x-www-form-urlencoded");
# Add the "Form fields" that are contained in the page...
  $req->content("IWPEProcessFlow.submitted.sequenceID=&logonUsername=m
+yhiddenusername&logonPassword=myhiddenpass&Logon=Login&hMsgUserNotFou
+nd=User with this name was not found.&hMsgInvalidPassword=Logon not v
+alid.&fromHomePage=default_B2B.htm");
# Spoof the referrer to make the site think it's a person
# from the login page...
  $req->referrer("http://www.eb2b.net/Templates/UM/RegisterCustomer/E2
+B_UMLogon.tem");
# Ok, now Perform Task
  $response = $browser->request($req);

if($response->is_success) {
# This is not working as it is posting in the 'ELSE' statement. It is 
+actually posting this in the "$response->content" field:

START $response->content
<HTML>
<!-- IWLoadCurrentCustomerLocale.tem -
      -- Copyright (c) 1997 InterWorld Corporation --> 
<HEAD>      
<META HTTP-EQUIV=REFRESH CONTENT="1;URL=/E2B_UMLogOn.process">
<TITLE>Refresh User Repository</TITLE>
</HEAD>

<BODY BGCOLOR="#ffffff">

<!--<A HREF="/E2B_UMLogOn.process">
<FONT SIZE="3" FACE="verdana, arial, helvetica">Please click here to c
+ontinue...</FONT></A>
-->
</BODY>
</HTML>
END $response->content

      [download]
      If not I'll try to figure out how to use the WWW::Mechanize module.

      Thank you for any further assistance you can be.

      Richard
[reply]
[d/l]
        I have put a TON of time into this and don't want to start at ground zero, unless I have to. Therefore, how would I get what I have already written to work?

        In short, no. Putting "TONs of time" into automating a web site without looking at a working connection will never work, or at least require inordinate amounts of "TONs of time".

        I recommend that you record a complete session with the actual browser, and that you don't fake a non-existing browser that has the UserAgent header Mozilla/5.0 - please look at the traffic sent by LWP and compare it to the traffic sent by your browser. Pretending to be a browser starts by sending the same data as the browser.

        I can only recommend to you to try a different route and let the code you alreday have lie still for a while. Try the approach of imitating a browser by mimicking exactly what it does by actually striving to send the data identically to what the browser would send.

        HTTP::Recorder (by leira) and WWW::Mechanize::Shell (by me) are two modules that will try to write WWW::Mechanize scripts for you, and maybe they will already suffice, but I cannot stress enough the point of comparing at what your script is sending against the working solution of what your browser is sending.

[reply]

Back to Seekers of Perl Wisdom