Undeleteable header in WWW::Mechanize

javahater has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I'm using WWW::Mechanize to duplicate various HTTP requests.

Since I want the request headers to be identical with my browser's headers, I modify them using add_header(). Everything works as intended except for some header named "TE" showing up in every request.


TE: deflate,gzip;q=0.3

I tried to get rid of it (see below) without success and I'm sort of confused why Mechanize enforces this header when my browser(s) don't even use it.

require WWW::Mechanize;

my $mech = WWW::Mechanize->new( agent => 'Mozilla/4.0 (compatible; MSI
+E 6.0; Windows NT 5.1)', timeout => 60, stack_depth => 1, autocheck =
+> 0);
$mech->add_header( Accept => 'text/xml,application/xml,application/xht
+ml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5');
$mech->add_header( "Accept-Language" => 'en-us;q=0.8,en;q=0.5,de-de;q=
+0.3');
$mech->add_header( "Accept-Charset" => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7
+');
$mech->add_header( Connection => 'keep-alive');
$mech->add_header( "Keep-Alive" => 10);

$mech->add_header( TE => undef);
[download]

I'd appreciate any help to get around this issue, preferably without hacking the module itself.

Thanks,
Björn

Comment on Undeleteable header in WWW::Mechanize Download Code

Replies are listed 'Best First'.
Re: Undeleteable header in WWW::Mechanize by aquarium (Curate) on Dec 19, 2007 at 02:02 UTC
TE = Transfer Encoding there is a delete_header function in WWW::Mechanize the hardest line to type correctly is: stty erase ^H	[reply]
Re^2: Undeleteable header in WWW::Mechanize by javahater (Initiate) on Dec 19, 2007 at 17:50 UTC
delete_header ( 'TE' ); does not do anything. Seems to be an internal header so it cannot be removed.	[reply]
Re: Undeleteable header in WWW::Mechanize by ikegami (Patriarch) on Dec 19, 2007 at 18:22 UTC
I've started looking into this. 1) It's not WWW::Mechanize adding that header. And even if it was, 2) The last thing WWW::Mechanize does before passing the request on to the underlying library is to process add_header directives, so `$mech->add_header( TE => undef);` would work if it was WWW::Mechanize adding that header. I'm off to check if it's something in libwww-perl, or it if it's an external source (such as a proxy).	[reply] [d/l]
Re^2: Undeleteable header in WWW::Mechanize by ikegami (Patriarch) on Dec 19, 2007 at 22:17 UTC
It took some searching, but I found that it's added by Net::HTTP at the request of LWP::Protocol::http. You can change the default by using `@LWP::Protocol::http::EXTRA_SOCK_OPTS = ( SendTE => 0, );` [download] You can use `local` to limit the scope of the change if you desire. `{ local @LWP::Protocol::http::EXTRA_SOCK_OPTS = ( SendTE => 0, ); ...do the request... }` [download] Untested.	[reply] [d/l] [select]
Re^3: Undeleteable header in WWW::Mechanize by javahater (Initiate) on Dec 20, 2007 at 21:08 UTC
works like a charm. I feel pretty stupid now, that'd have been the last place I would have looked. That also fixed an issue with the Connection header being set to "keep-alive, close" since it's possible to set a proper Keep-Alive header like this: `local @LWP::Protocol::http::EXTRA_SOCK_OPTS = ( SendTE => 0, KeepAlive => 1 );` [download] Thank You!	[reply] [d/l]