CGI::Carp fatalsToFile

Bod has asked for the wisdom of the Perl Monks concerning the following question:

I'm looking for some debugging help please
Ideally I would like something like a fatalsToFile equivalent to the CGI::Carp fatalsToBrowser.

I have what appears to be the same issue on two different websites. They are both on the same shared hosting but otherwise totally independent.

The most critical issue is with a script that gets booking information from OTAs, Online Travel Agents (Airbnb, Booking.com, etc) using their API to provide an ICS file. Our booking platform checks this information as well as our own booking database to show customers our availability. Our script caches the ICS file for 30 minutes. This script has an API that our booking platform calls to get a simple text file which lists the property number and either 'available' or 'booked'. When a customer commits to book, the API is called again but with a force parameter to bypass the cache.

Except the API returns a 500 error when called from the booking platform. But I cannot replicate it when calling the script using a web browser.

Here's the code we had working. I know there are some bad practices in there but this is legacy code and will get improved in due course. The important thing is that is was working fine until recently and I am not aware of anything that could have caused the change:

if (!defined @avail) {
    foreach (split /\n/, &get("https://$ENV{'HTTP_HOST'}/cgi-bin/booki
+ng.pl?command=check&st=$data{'st'}&ed=$data{'ed'}&force=$data{'force'
+}", $availability);) {
        chomp;
        my ($k, $v) = split/,/;
        $avail[$k] = $v;
    }
}
[download]

The &get is from LWP::Simple.
To try and understand what was going on, I changed the code to this:

my $availability = LWP::Simple::get("https://$ENV{'HTTP_HOST'}/cgi-bin
+/booking.pl?command=check&st=$data{'st'}&ed=$data{'ed'}&force=$data{'
+force'}");

print "->$availability<-\n";

if (!defined @avail) {
    foreach (split /\n/, $availability) {
        chomp;
        my ($k, $v) = split/,/;
        $avail[$k] = $v;
    }
}
[download]

and found that $availability was empty.

my $availability;
my $response = LWP::Simple::getstore("https://$ENV{'HTTP_HOST'}/cgi-bi
+n/booking.pl?command=check&st=$data{'st'}&ed=$data{'ed'}&force=$data{
+'force'}", $availability);
print qq[("https://$ENV{'HTTP_HOST'}/cgi-bin/booking.pl?command=check&
+st=$data{'st'}&ed=$data{'ed'}&force=$data{'force'}")\n];
print "->$availability<-\n";
print "$response\n";
[download]

This shows that the response code is 500.
Copying the printed URL into a web browser gives the text output that I would expect.

I have checked the error logs provided to me by cPanel and nothing shows up at all.

Any ideas why a script would return a 500 error when called by LWP::Simple get but not when called by Chrome?
There is nothing in the script that checks the user agent and it was previously working.

I have a similar, but intermittent problem on another site. This time the script is called from the browser using the fetch API. The script returns a JSON object. Again, the script returns a 500 error some of the time but works as predicted most of the time (unlike above which fails every time). Again I am at a loss how to debug this as I cannot catch the errors to find what or where they are.

This is the Javascript from the Template file that calls the Perl API which generates the error:

fetch('/api/', {
        method: 'POST',
        body:   'a=[% token %]&u=[% user %]&q=getWays&' + request,
    }).then(response => response.json())
      .then(data => {
        document.getElementById('mapmsg').innerHTML = '';
        if (data[0][0].status == 'success') {
            plotWays(data);
        } else {
            document.getElementById('timeoutbox').style.display='block
+';
        }
    })
    .catch(error => {
        console.log(error.message);
});
[download]

If I had any hair, I would be pulling it out...

Comment on CGI::Carp fatalsToFile Select or Download Code

Replies are listed 'Best First'.
Re: CGI::Carp fatalsToFile by hippo (Bishop) on Oct 05, 2021 at 18:38 UTC
Any ideas why a script would return a 500 error when called by LWP::Simple get but not when called by Chrome? Was it OK up until last Thursday afternoon by any chance? If so, it is the expiry of the old DST Root cert which is the most likely cause. You'll have to update the CA store on the server - check the version of Mozilla::CA installed. 🦛	[reply]
Re^2: CGI::Carp fatalsToFile by Fletch (Bishop) on Oct 05, 2021 at 19:31 UTC
Seconding this as something to check (on RHEL it's your "ca-certificates" RPM). Even if you're hitting somewhere "big" you wouldn't think would be using a free cert service because we had someone at $work start getting gripes from `wget` from the NY FED's site. The cake is a lie. The cake is a lie. The cake is a lie.	[reply] [d/l]
Re^2: CGI::Carp fatalsToFile by Bod (Parson) on Oct 05, 2021 at 21:19 UTC
Thanks hippo - I had forgotten about the old DST Root certificate issue... It gave me an avenue to explore but alas, that's not it. To test I have edited the `.htaccess` so that it does not enforce HTTPS. Then I have changed the code slightly to remove the HTTPS: `my $availability; my $response = LWP::Simple::getstore("http://$ENV{'HTTP_HOST'}/cgi-bin +/booking.pl?command=check&st=$data{'st'}&ed=$data{'ed'}&force=$data{' +force'}", $availability); print qq[("http://$ENV{'HTTP_HOST'}/cgi-bin/booking.pl?command=check&s +t=$data{'st'}&ed=$data{'ed'}&force=$data{'force'}")\n]; print "->$availability<-\n"; print "$response\n";` [download] This still behaves the same. `$availability` is empty and `$response` is 500. I have checked that pasting the URL into a browser still works and doesn't change to HTTPS. Any other ideas of anything I can try to attempt to debug the issue?	[reply] [d/l] [select]
Re^2: CGI::Carp fatalsToFile by Bod (Parson) on Oct 08, 2021 at 18:41 UTC
You'll have to update the CA store on the server - check the version of Mozilla::CA installed. It seems you may be right after all hippo! Is updating the CA store on the server something I can do on shared hosting or do I have to get the server admins to do it? I am able to install new certificates and historically I have had to until, earlier this year, they arranged for Lets Encrypt to automagically renew them. The Mozilla::CA version is 20120823 I assume that's a date - a very old date! UPDATE: I've answered my own question - I can sort of update the CA store... cPanel has two places for modules. Server wide modules and user level modules. I can, and have, done the update at a user level. To use this version instead of the server wide module I have to have `use cPanelUserConfig;` before I `use` any modules that rely on the CA store being up to date.	[reply] [d/l] [select]
Re^3: CGI::Carp fatalsToFile by pryrt (Abbot) on Oct 08, 2021 at 18:58 UTC
Is updating the CA store on the server something I can do on shared hosting It probably depends on the host. My shared hosting had 20130114 installed; when I used their interface to install Mozilla::CA, it now reports 20211001. Please note that the Lets Encrypt certificates that are automagically renewed are the certificates that allow people to securely connect to your shared host under your domain name(s). That's separate from the CA store on the server, which is the list of certificates that your perl scripts will use to determine if a machine they connect to have a valid certificate or not.	[reply]
Re^4: CGI::Carp fatalsToFile by Bod (Parson) on Oct 08, 2021 at 19:04 UTC
Re: CGI::Carp fatalsToFile by tangent (Parson) on Oct 06, 2021 at 02:56 UTC
You could try putting this at the top of the script: `use CGI::Carp qw(set_die_handler); BEGIN { sub handle_errors { my $msg = shift; print "Content-Type: text/html\n\n"; print "$msg"; } set_die_handler(\&handle_errors); }` [download] This will change the server's response code from 500 to 200 and would give you content like: `[Wed Oct 5 23:53:36 2021] test.cgi: Died at /web/test.cgi line 15.` [download] However, the fact that the script you are calling with LWP::Simple does not appear in the logs would suggest that it is not actually being called. I would not discount the role of user-agent - non-standard user-agents can be blocked to prevent web scraping. Maybe something at the server level was updated or reconfigured by your hosting company? To test you will need to use LWP::UserAgent directly. `use LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->agent('Mozilla/5.0'); # play with this string my $response = $ua->get('http://localhost/web/test.cgi'); if ($response->is_success) { print $response->decoded_content; } else { print $response->status_line; }` [download]	[reply] [d/l] [select]
Re^2: CGI::Carp fatalsToFile by Bod (Parson) on Oct 06, 2021 at 16:25 UTC
Thanks for the very helpful reply You could try putting this at the top of the script: `use CGI::Carp qw(set_die_handler); BEGIN { sub handle_errors { my $msg = shift; print "Content-Type: text/html\n\n"; print "$msg"; } set_die_handler(\&handle_errors); }` [download] I'm going to write some test code so that I can properly test what is going on. It will probably have to wait until the weekend though. But this code has been a little confused! Is `set_die_handler` a method of CGI::Carp and `set_die_handler(\&handle_errors);` in the `BEGIN` block is overriding it? Or is something different happening here?	[reply] [d/l] [select]
Re^3: CGI::Carp fatalsToFile by tangent (Parson) on Oct 06, 2021 at 17:34 UTC
Is set_die_handler a method of CGI::Carp and set_die_handler(\&handle_errors); in the BEGIN block is overriding it? `set_die_handler` is a method of CGI::Carp, but its purpose is to allow you to specify your own function to be executed "any time a script calls "die", has a syntax error, or dies unexpectedly at runtime". See the docs for more.	[reply] [d/l]
Re: CGI::Carp fatalsToFile by bliako (Monsignor) on Oct 06, 2021 at 11:07 UTC
`Any ideas why a script would return a 500 error when called by LWP::Simple get but not when called by Chrome?` Be aware that LWP::UserAgent creates and emits its own 500 errors on certain occasions, see Re^2: HTTP error response code 500 using LWP::UserAgent on one site, but not on any other by afoken. So, what you are seeing may be coming from LWP's handling of the transaction and not the server (running your cgi-script). Two differences between Chrome and your LWP test script can be the response/connection timeout and the user-agent string, in addition to certificate-related problems which you seem to have excluded. The intermittent behaviour you mentioned from the 2nd script can again be cause by short timeouts, e.g. when the script fetches the cached file it takes very short time and is within the timeout. But when the cache expires, it will take significantly longer and can fail the timeout. But even if that script does not use a cache, try setting a long timeout in both scripts (the cgi and the test) and see. When I use LWP for purposes other than playing around, I want to keep as much control as possible: I always use LWP::UserAgent. Finally, there's also LWP::Curl with same API which you can use as an alternative, though LWP::UserAgent never failed me. (I also prefer to use a host with unix CLI so as to avoid hair pulling, but I said that before already...) bw, bliako	[reply] [d/l]
Re: CGI::Carp fatalsToFile by pryrt (Abbot) on Oct 05, 2021 at 21:28 UTC
Since it's not https-related, you're probably back to wanting a `fatalsToFile` equivalent. Looking at CGI::Carp source, it appears they just use a `__DIE__` signal handler; so you should be able to write such a handler that writes to the desired file rather than to STDOUT or STDERR.	[reply] [d/l] [select]
Re^2: CGI::Carp fatalsToFile by Bod (Parson) on Oct 05, 2021 at 21:37 UTC
so you should be able to write such a handler... I love your faith in my abilities! I'm not so sure...but, you are right, the CGI::Carp source doesn't look especially complex. Perhaps there is another big step in the stairway of learning directly in front of me.	[reply]
Re^3: CGI::Carp fatalsToFile by afoken (Chancellor) on Oct 05, 2021 at 22:01 UTC
Just omitting CGI::Carp and just using plain old Carp should do the trick. Carp behaves nicely and writes all errors to STDERR, where they belong. The web server collects STDERR output and writes that to its error log. Problem solved, even when multiple CGIs are running concurrently (which might create nasty races when trying to write to a common log file at the CGI level). That also works when your CGI has a really bad day and just crashes before Carp or CGI::Carp can do their work. The only remaining problem might be the hoster: Some really cheap hosters prevent you from reading your server's error log. That really sucks, and cannot be solved completely at the application (CGI) layer. If that is the case, consider changing the hoster or upgrading to a contract with access to the error log. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]
Re^4: CGI::Carp fatalsToFile by Bod (Parson) on Oct 05, 2021 at 23:17 UTC
Re^2: CGI::Carp fatalsToFile by Bod (Parson) on Oct 08, 2021 at 18:29 UTC
Since it's not https-related, you're probably back to wanting a fatalsToFile equivalent It turns out that it is https-related... There were two red herrings. Firstly the `.htaccess` uses a 301 redirect that was being cached and also the `getstore` method of LWP::Simple is not doing what I expected. `my $availability; my $response = LWP::Simple::getstore("http://$ENV{'HTTP_HOST'}/cgi-bin +/booking.pl?command=check&st=$data{'st'}&ed=$data{'ed'}&force=$data{' +force'}", $availability); print "->$availability<-\n"; print "$response\n";` [download] Shows `$availability` as empty even though the server is responding with 200 code. So, now I know that it is an HTTPS issue, I need to find a way to fix it...	[reply] [d/l] [select]
Re^3: CGI::Carp fatalsToFile by pryrt (Abbot) on Oct 08, 2021 at 18:36 UTC
Are you sure you're using `getstore` correctly? Per the LWP::Simple docs: *getstore* `my $code = getstore($url, $file)` Gets a document identified by a URL and stores it in the file. The return value is the HTTP response code. getstore desires a URL and a filename. Your code snippet shows `$availability` as being undefined.	[reply] [d/l] [select]
Re^4: CGI::Carp fatalsToFile by Bod (Parson) on Oct 08, 2021 at 18:47 UTC
Re^5: CGI::Carp fatalsToFile by pryrt (Abbot) on Oct 08, 2021 at 19:02 UTC

Back to Seekers of Perl Wisdom