sedusedan has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

A few weeks ago, my module [Finance::Bank::ID::BCA] started failing to connect to the bank site https://ibank.klikbca.com/ with a Connection Reset error. The module uses [WWW::Mechanize] (which is a subclass of LWP::UserAgent). A quick check with curl seems to indicate that the server now denies HTTP/1.0 protocol: 

% curl -v https://ibank.klikbca.com/ ; # success
% curl --http1.0 -v https://ibank.klikbca.com/ ; # fails

[download]

Using wget also fails.

But checking with Gepok (gepok.client_protocol) indicates that wget as well as LWP already uses HTTP/1.1 instead of HTTP/1.0 to the server.

Also worth noting, don't know if this is correct, my Debian Squeeze wget seems to be only using TLSv1 (from playing with --secure-protocol option) while curl seems to use SSLv3 (looking at -v output).

Any pointer is appreciated on how to make LWP work with this site. As a last resort I might use curl (via [WWW::Curl], for example) but I'll probably need to replace WWW::Mechanize too, which will be painful.

Replies are listed 'Best First'.
Re: Investigating connection reset with LWP to an https site
by Khen1950fx (Canon) on Mar 05, 2012 at 19:40 UTC
    I tried to install your module, but I ran into a problem with Crypt::SSLeay. The problem for me is that, out of my dozen perls, they're dynamic except for two that are static. It seems that Crypt::SSLeay simply will not install on my dynamic perls; however, it will install without problems on my static perls. Note that LWP::Protocol::https must be installed in order to use https. You might want to add it as a dependency.
[reply]
      Good catch on adding LWP::protocol::https as dep (it was split into its separate dist in 6.x), thanks.
[reply]
Re: Investigating connection reset with LWP to an https site
by jethro (Monsignor) on Mar 05, 2012 at 17:11 UTC

    So if LWP already uses HTTP/1.1 why do you still think this is the problem? Maybe when the bank changed its web handling they changed other parameters too

[reply]
      So if LWP already uses HTTP/1.1 why do you still think this is the problem? Maybe when the bank changed its web handling they changed other parameters too

      That's what I'm trying to find out :) I'm getting a Connection Reset error, so this is a network problem instead of HTML/JavaScript. Besides, I can still copy paste the HTML source code that I get from my browser and parse it with the module's parse_statement(), so that part is not having problem.

[reply]
Re: Investigating connection reset with LWP to an https site
by bcarroll (Pilgrim) on Mar 05, 2012 at 18:28 UTC
    Have you tried adding the" MIME_Version" header?

    Something like this:

    $mech->add_header( 'MIME_Version' => '1.0' );

    [download]

    From: http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html
    MIME version "1.0" is the default for use in HTTP/1.1.

[reply]
[d/l]

      No mainstream browser has ever included a MIME-Version header in its requests. Thus no websites are likely to pay any attention to it whatsoever.

[reply]
[d/l]
Re: Investigating connection reset with LWP to an https site
by Gangabass (Vicar) on Mar 05, 2012 at 23:46 UTC
    I recommend you login via browser with Javascript disabled and check all params sended by your browser to the target site (via Firefox's HTTPFox or Opera's Dragonfly).

    Next you just need to send all these params from your script (Referer, cursor click x and y etc.)

    Thanks.
    Roman

[reply]
      To restate my problem: I'm getting a connection reset error instead of the site changing HTML/layout/design/JavaScript (it doesn't, BTW). The module can't even retrieve a single page, like the front page or the login page, from the site. So I'm still guessing it has to do with the network layer like LWP or Crypt::SSLeay.
[reply]

        RESET can happen for lots of different reasons. They could even be suddenly breaking the connection after they see your user agent string.

        I'd get more information by capturing packets with wireshark to compare the different scenarios.

        - tye        

[reply]

Back to Seekers of Perl Wisdom