Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi wise monks.... I am having an issue with gathering and reading a single page... I am not sure what is going on, but a very simple script would be: 
#!/usr/bin/perl -w
use strict;
use warnings;
use LWP::Simple;

my $initial_url = 'http://www.sciencedirect.com/science/search';

my $result = get($initial_url);

[download]
I get an Error code 500 when I do this, but no issues if I go through a web browser? any insight would really be appreciated!

Replies are listed 'Best First'.
Re: Having problems getting a single page
by ww (Archbishop) on Feb 04, 2017 at 16:26 UTC

    One wise step often: take a look at the header -- a fragment from your target follows:

    <iframe src="javascript:false" title="" style="width: 0px; height: 0px
+; border: 0px none; display: none;"></iframe>
<script type="text/javascript" async="" src="//cdn.optimizely.com/js/2
+04774041.js"></script>
....

    [download]

    But, as it turns out, there are several threads here about similar problems, of which, this is one: Request by LWP Useragent refused by the web server but not by others.

    That was found with Super Search -- a feature here that you'll often find helpful if you're careful about picking your search_terms.


    Spirit of the Monastery


[reply]
[d/l]
[select]
      Hi, Thank you for your help, I will try to look into this.
[reply]
Re: Having problems getting a single page
by Anonymous Monk on Feb 04, 2017 at 21:26 UTC
    I am sorry, I have tried looking at the aforementioned pages, and I am not making any head way, it is likely my own ignorance. Are you saying that I am not setting the header correctly with the User Agent so am I a not getting the page? i.e. this javascript on the page: 
    <script>(function() {var scriptTag = document.createElement('script');
+ 
scriptTag.type = 'text/javascript';scriptTag.async = true;
scriptTag.src = '//cdn.optimizely.com/js/204774041.js';
var head = document.getElementsByTagName('head')[0];
head.parentNode.insertBefore(scriptTag, head);
})();function optimizelyTimeout() {
window.optimizely = window.optimizely|| []; 
if(!window.optimizely.data) {
window.optimizely.push("timeout"); } }
setTimeout(optimizelyTimeout, 1000); </script>

    [download]
    Sorry for the stupid questions, hoping that if I can get a nudge in the right direction, perhaps a bit more than just search for the right term and you will get the answers. Maybe if you could help show me the right ideas that I should be searching for. Thank you in advance.
[reply]
[d/l]

Back to Seekers of Perl Wisdom