Error when calling $mechanize->content;

Jacob_Kold has asked for the wisdom of the Perl Monks concerning the following question:

How come this webpage gives me an internal server error when I try to scrape it with Mechanize

my $mech = WWW::Mechanize->new();
$mech->get( https://www.aabenraa.dk/politik-og-dialog/politiske-udvalg
+/arbejdsmarkedsudvalget/medlemmer );
my $html_udv = $mech->content;
[download]

And some html containing the line: "Object reference not set to an instance of an object" When I try to scrape the same webpage with UserAgent?

 
my $ua  = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 },);
my $res = $ua->get('https://www.aabenraa.dk/politik-og-dialog/politisk
+e-udvalg/arbejdsmarkedsudvalget/medlemmer');
my $html = $res->content;
[download]

What can I do to get the HTML of the webpage and not get any errors?

Comment on Error when calling $mechanize->content; Select or Download Code

Replies are listed 'Best First'.
Re: Error when calling $mechanize->content; by marto (Cardinal) on Apr 18, 2018 at 14:15 UTC
Your first example is missing single quotes for the working URL, your second example has a typo in the URL which returns 'This page was not found' (Google translated). `https://www.aabenraa.dk/politik-og-dialog/politiske-udvalg/arbejdsmark +edsudvalget/medlemmer https://www.aabenraa.dk/politik-og-dialog/politisk-udvalg/arbejdsmarke +dsudvalget/medlemmer` [download] Even when setting a valid UA string, I get the ASP.net error 'NullReferenceException: Object reference not set to an instance of an object.]', and some debug info best kept away from prying eyes. There's certainly something making it difficult to screenscrape. Perhaps they don't want you to.	[reply] [d/l]
A reply falls below the community's threshold of quality. You may see it by logging in.