hoanglearning has asked for the wisdom of the Perl Monks concerning the following question:

My program don't get URL response in search for capacitor part from DIGI-key. I received good response when URL is typed in directly on Google Chrome browser. When running my program, it shows the HTTP status of 302 without URL redirect and no response is received. Please help to point out what wrong with my codes. Thanks so much in advance

 
use strict;
  use warnings;
  use LWP;
  my $browser = LWP::UserAgent->new;
     $browser->env_proxy;

  my $url = 'http://www.digikey.com/product-search/en?KeyWords=C0603C4
+74K4PACTU&WT.z_header=search_go';

  my $response = $browser->get( $url ); 
print "HTTP status: ", $response->code( ), "\n";

  unless ($response->is_success) {
     print "no response\n";
  }
  my $response_content = $response->content;
  print "response_content= $response_content\n";

[download]

The modified codes from Sathishkumar resolved this problem. Sathishkumar, thanks verymuch. I got your codes and it's working. I still need to learn the line "$cookie->extract_cookies($res);" to see what it exactly doing. I also like to thanks Perlmonk and MidLifeXis. I can't not get this problem solved without your help.

Replies are listed 'Best First'.
Re: Use LWP get no URL response
by MidLifeXis (Monsignor) on Oct 02, 2015 at 13:16 UTC

    Look at the Location header in the response. I see the Location header in my curl -v query.

    Never mind - the redirect location is pointing to an error page.

    Perhaps lwp (and curl) are not allowed clients? Compare the headers sent to find the differences. Check the ToS on Digi-Key's site, as they may not allow mechanized searches.

    You may not use any robot, scraper, spider, or other automated means to access or gather the Content from the Site, or mirror, co-brand, or frame the home page or any other pages of the Site on any other website or web page. For purposes of these Terms of Use, “co-branding” means to display a name, logo, trademark, or other means of attribution or identification of any party in such a manner as is reasonably likely to give a user the impression that such other party has the right to display, publish, or distribute the Site or its Content. Competitors and third party aggregators may not connect "deep links" to the Site, i.e., create links to this site that bypass the home page or other parts of the Site without the written permission of Digi-Key. This prohibition is not intended to restrict the non-commercial activities of individuals.

    --MidLifeXis

[reply]
[d/l]
[select]
Re: Use LWP get no URL response
by Sathishkumar (Scribe) on Oct 02, 2015 at 07:04 UTC
    Hi, I think you miss accept type in the code. here I attached correct code for your help. 
    use strict;
use HTTP::Cookies;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(show_progress=>1);
$ua->agent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/2010010
+1 Firefox/26.0");
$ua->timeout(30);
$ua->cookie_jar({});
my $filename ="indiegogo";
my $cookie_file = $filename."_cookie.txt";
unlink($cookie_file);
my $cookie = HTTP::Cookies->new(file=>$cookie_file,autosave=>1);
$ua->cookie_jar($cookie);
my $url = 'http://www.digikey.com/product-search/en?KeyWords=C0603C474
+K4PACTU&WT.z_header=search_go';
my $req = HTTP::Request->new(GET=>$url); 
$req->header("Accept"=>"text/html,application/xhtml+xml,application/xm
+l;q=0.9,*/*;q=0.8"); 
$req->header("Content-Type"=>"application/x-www-form-urlencoded"); 
my $res = $ua->request($req); 
$cookie->extract_cookies($res); 
$cookie->save; 
$cookie->add_cookie_header($req); 
my $code = $res->code(); 
print $code,"\n"; 
if($code =~ m/50/is) 
{ 
    sleep 1000; 
    goto REPEAT; 
}
my $content = $res->content();
open sr,">digikey.html";
print sr $content;
close sr;

    [download]
    Thanks, Sathish
[reply]
[d/l]
Re: Use LWP get no URL response
by Anonymous Monk on Oct 02, 2015 at 01:42 UTC
[reply]

Back to Seekers of Perl Wisdom