www::mechanize problem with url / redirect

McSvenster has asked for the wisdom of the Perl Monks concerning the following question:

Dear wise monks,

I am - have to - working with really bad software... Even worse, the software is behind a reverse proxy, so redirects are really complex. I am trying to write some test scripts with www::mechnize. As the redirects happen through javascript I parse the content to follow the redirections:

$agent->get($url);
$agent->form_number(1);
$agent->field('vl(freeText0)', $suchwort);
$agent->click();
$agent->content =~ m/\'\/goto\/(.*)\'/;
print "Der Link: $1\n\n";
$agent->get( $1);
[download]

As You can see I print the resulting link for my analysis. Unfortunately my agent seems not to use the link provided in $1. Here the output:

Der Link: http://www.library.ethz.ch:80/search/action/login.do?afterPD
+S=true&vid=DADS&vid=DADS&dscnt=1&targetURL=http%3A%2F%2Fwww.library.e
+thz.ch%2Fsearch%2Faction%2Fsearch.do%3Fdscnt%3D0&frbg=&tab=default%5F
+tab&dstmp=1356212616435&srt=rank&ct=search&mode=Basic&dum=true&indx=1
+&vl%28freeText0%29=Ajax&cmslang=ger-DE&fn=go&pds_handle=GUEST

Error GETing http://www.library.ethz.ch/search/action/http%3A%2F%2Fwww
+.library.ethz.ch%2Fsearch%2Faction%2Fsearch.do%3Fdscnt%3D0dscnt=2&frb
+g=&tab=default%255Ftab&dstmp=1356212617376&srt=rank&ct=search&mode=Ba
+sic&dum=true&indx=1&vl%28freeText0%29=Ajax&fn=go&vid=DADS&cmslang=ger
+-DE&fromLogin=true&fromLogin=true: Not Found at mech_auto.pl line 24
[download]

I have no idea what is happening here! It looks like there is a link and a redirect mixed up. The first link works without a problem in a browser and redirects to:

http://www.library.ethz.ch/search/action/search.do?&dscnt=1&frbg=&tab=
+default_tab&dstmp=1356213423770&srt=rank&ct=search&mode=Basic&dum=tru
+e&indx=1&fromLogin=true&vl(freeText0)=Ajax&vid=DADS&fn=search&cmslang
+=ger-DE
[download]

Can anyone of You please send some enlightenment? Thanks, MC

Comment on www::mechanize problem with url / redirect Select or Download Code

Replies are listed 'Best First'.
Re: www::mechanize problem with url / redirect by Anonymous Monk on Dec 22, 2012 at 22:08 UTC
Try using `->get("$1");` Turn on `->show_progress(1);` and `->add_handler("request_send", sub { shift->dump; return });` and `->add_handler("response_done", sub { shift->dump; return });` and analyze the results	[reply] [d/l] [select]
Re^2: www::mechanize problem with url / redirect by McSvenster (Novice) on Dec 22, 2012 at 22:27 UTC
wow, that was fast - thanks a lot! using `("$1")` made no difference. The other hints will take time to analyse for me. Again Thank You!	[reply] [d/l]
Re^3: www::mechanize problem with url / redirect by McSvenster (Novice) on Jan 03, 2013 at 06:55 UTC
Just for the records: I added the code like this: `$agent->show_progress(1); $agent->add_handler("request_send", sub { shift->dump; return }); $agent->add_handler("response_done", sub { shift->dump; return }); $agent->get("$1"); print $agent->content;` [download] It gave me a bunch of feedback like: (no content) HTTP/1.1 301 Moved Permanently Connection: close So I found the redirects and could break down the problem. Thanks a lot again! Mc	[reply] [d/l]