Krambambuli has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

I'm searching your help while trying to overcome an issue with a script using Net::Curl::Multi in an Amazon Linux 2023 instance.

At first run, I hit the wall and get a trace like
$ ./04-curl-multi-debug.pl TRACE 1 TRACE 2 TRACE 3 TRACE 4 GET_ONE 1 GET_ONE 2 GET_ONE 3 $$self: 1 GET_ONE 4 GET_ONE 6 GET_ONE 3 $$self: 1 GET_ONE 4 callback function is not set Operation was aborted by an application callbackTRACE 6: 'Operation wa +s aborted by an application callback'
Subsequent runs will do their job OK - for a short while. Then again a "callback function is not set"... And then again a number of runs will do OK. The script I've tried to track down the issue is here:
#!/usr/bin/perl package Multi::Simple; use strict; use warnings; use Net::Curl::Multi; use base qw(Net::Curl::Multi); # make new object, preset the data sub new { my $class = shift; my $active = 0; return $class->SUPER::new( \$active ); } # add one handle and count it sub add_handle($$) { my $self = shift; my $easy = shift; $$self++; $self->SUPER::add_handle( $easy ); } # perform until some handle finishes, does all the magic needed # to make it efficient (check as soon as there is some data) # without overusing the cpu. sub get_one($) { my $self = shift; print "GET_ONE 1 \n"; if ( my @result = $self->info_read() ) { $self->remove_handle( $result[ 1 ] ); return @result; } print "GET_ONE 2 \n"; while ( $$self ) { print "GET_ONE 3 \$\$self: $$self\n"; my $t = $self->timeout; if ( $t != 0 ) { $t = 10000 if $t < 0; my ( $r, $w, $e ) = $self->fdset; select $r, $w, $e, $t / 1000; } print "GET_ONE 4 \n"; my $ret = $self->perform(); if ( $$self != $ret ) { $$self = $ret; if ( my @result = $self->info_read() ) { $self->remove_handle( $result[ 1 ] ); return @result; } print "GET_ONE 5 \n"; } print "GET_ONE 6 \n"; }; print "GET_ONE 7 \n"; return (); } 1; ############################################### ############################################### ############################################### package main; #use strict; use warnings; BEGIN { push @INC, './'; } use Data::Dumper; #use Multi::Simple; use Net::Curl qw(:constants); use Net::Curl::Easy qw(:constants); use Net::Curl::Multi qw(:constants); use Net::Curl::Share qw(:constants); #print "libcurl version: ", Net::Curl::version(), "\n"; # my $vi = Net::Curl::version_info(); # unless ( $vi->{features} & CURL_VERSION_SSL ) { # die "SSL support is required\n"; # } #print "Version Info: ", Dumper( $vi ); #use Net::Curl; #my $info = Net::Curl::version_info(); #print "libcurl version: $info->{version}\n"; #print "ares_num: $info->{ares_num}\n"; #print "features: $info->{features}\n"; #exit; sub easy { my $uri = shift; my $share = shift; #require Net::Curl::Easy qw(:constants); my $easy = Net::Curl::Easy->new( { uri => $uri, body => '' } ) +; $easy->reset; $easy->setopt( Net::Curl::Easy::CURLOPT_VERBOSE(), 1 ); $easy->setopt( Net::Curl::Easy::CURLOPT_SSL_SESSIONID_CACHE, 0 + ); $easy->setopt( Net::Curl::Easy::CURLOPT_DNS_USE_GLOBAL_CACHE, +0); $easy->setopt( Net::Curl::Easy::CURLOPT_URL(), $uri ); $easy->setopt( Net::Curl::Easy::CURLOPT_WRITEHEADER(), \$easy->{headers} ); $easy->setopt( Net::Curl::Easy::CURLOPT_FILE(), \$easy->{body} ); #$easy->setopt( Net::Curl::Easy::CURLOPT_SHARE(), $share ); # This wasn't needed prior to curl 7.67, which changed the int +erface # so that an easy that uses a cookie-share now requires an exp +licit # cookie-engine enable to use cookies. Previously the easy's u +se of # a cookie-share implicitly enabled the easy's cookie engine. #$easy->setopt( Net::Curl::Easy::CURLOPT_COOKIEFILE(), q<> ); return $easy; } my $multi; eval { $multi = Multi::Simple->new(); print "TRACE 1\n"; my $dummy = Net::Curl::Easy->new; $dummy->setopt(Net::Curl::Easy::CURLOPT_URL, 'http://example.com') +; #$dummy->setopt(Net::Curl::Easy::CURLOPT_WRITEFUNCTION, sub { retu +rn length($_[0]); }); if (not ref $dummy) { die "Cannot create easy object...?!\n"; } print "TRACE 2\n"; $multi->add_handle( $dummy ); print "TRACE 3\n"; $multi->wait(10000); print "TRACE 4\n"; my @results = $multi->get_one(); if (scalar( @results ) == 0) { die "Some ERROR occured in get_one!\n"; } my ($msg, $easy, $result) = @results; print "TRACE 5 Msg: ", Dumper( $msg ), "easy: ", Dumper( $easy ), +"Result: " , Dumper( $result ); #Net::Curl::Easy::strerror( $result ), "\n"; if ($result != Net::Curl::Easy::CURLE_OK) { my $message = Net::Curl::Easy::strerror( $result ); print "Result as string: '$message'\n"; } }; if ($@) { print "TRACE 6: '$@'"; die $@; } print "NO ERROR, all OK\n"; exit;
The code runs fine on an older Amazon Linux 2 instance. Any idea on how a work-around might look like? Thank you!

Replies are listed 'Best First'.
Re: Net::Curl::Multi on Amazon Linux 2023 instance
by Corion (Patriarch) on Nov 20, 2024 at 20:27 UTC

    Searching Net::Curl::Multi for callback function is not set returns a function of Net::Curl::Multi that raises this error, because some callback was not set. But it does not tell us, which callback it wanted to call.

    As the problem is intermittent, I would assume some timeout or error condition. Maybe you can diagnose that using Wireshark or some other network sniffer, or find another way to set up all kinds of callback handlers to see which one placates the behaviour of Net::Curl or Net::Curl::Multi.

      Thank you.

      It is for sure a DNS related issue, as strace and the proper use of verbose debugging (ineffective in the buggy debug code that I have posted) show.

      I am suspecting that there is an issue with Curl.xs and AsyncDNS, and that Curl.xs (v. 0.56) does _not_ provide a way to set CURLOPT_RESOLVER_START_FUNCTION, which might be necessary for and is available in the underlying libcurl (v. 8.5.0).

      I've emailed a question to the maintainer of Net::Curl::Multi, maybe he'll answer.

      Hope so, as I am not at all comfortable with the .xs stuff.

      I'm curious and have NO IDEA how I could find out which exactly the missing callback is.

Re: Net::Curl::Multi on Amazon Linux 2023 instance
by bliako (Abbot) on Nov 21, 2024 at 11:44 UTC

    I can confirm that this happened to a recently updated linux box once (the first time). And then 30 times it does not happen. I found that a way to make the problem appear is to restart the network service.

    Following Corion's investigation, the perl_curl_call() function is then typedef'ed into PERL_CURL_CALL which can be found being called in a few places: find . -type f -exec grep -H -i 'perl_curl_call' \{\} \; which you can investigate manually.

    Additionally, add code to print a stacktrace each time said function is called. I used https://stackoverflow.com/questions/105659/how-can-one-grab-a-stack-trace-in-c (the C part) or try https://gist.github.com/mooware/1198160. Unfortunately in my case the stacktrace does not contain function names but relative addresses from the object file. Theoretically, one can translate these to function names with other tools that linux provides generously, e.g. addr2line, objdump.

    This is the patch to the modified Curl.xs for printing a stacktrace:

    --- ../oo/Net-Curl-0.56/Curl.xs 2022-08-16 08:44:34.000000000 +0300 +++ Curl.xs 2024-11-21 12:11:53.087671081 +0200 @@ -4,6 +4,7 @@ * Perl interface for libcurl. Check out the file README for more inf +o. */ +#include <execinfo.h> /* * Copyright (C) 2000, 2001, 2002, 2005, 2008 Daniel Stenberg, Cris B +ailiff, et al. * Copyright (C) 2011-2015 Przemyslaw Iskra. @@ -278,6 +279,14 @@ SV *olderrsv = NULL; int method_call = 0; +void* callstack[128]; +int frames = backtrace(callstack, 128); +char** strs = backtrace_symbols(callstack, frames); +for (int I = 0; I < frames; ++I) { + printf("STACKTRACE: %s\n", strs[I]); +} +free(strs); + if ( ! cb->func || ! SvOK( cb->func ) ) { warn( "callback function is not set\n" ); return -1;

    This is the stacktrace for my situation, not very helpful without function names though:

    TRACE 6: 'Operation was aborted by an application callback'STACKTRACE: + blib/arch/auto/Net/Curl/Curl.so(+0x10d09) [0x7f5f16740d09] STACKTRACE: blib/arch/auto/Net/Curl/Curl.so(+0x11d7b) [0x7f5f16741d7b] STACKTRACE: /usr/lib64/libcurl.so.4(+0x51b84) [0x7f5f166cdb84] STACKTRACE: /usr/lib64/libcurl.so.4(+0x5542b) [0x7f5f166d142b] STACKTRACE: /usr/lib64/libcurl.so.4(curl_multi_perform+0x105) [0x7f5f1 +66d4265] STACKTRACE: blib/arch/auto/Net/Curl/Curl.so(+0xb834) [0x7f5f1673b834] STACKTRACE: perl() [0x4fdd6c] STACKTRACE: perl(Perl_runops_standard+0x13) [0x4f4023] STACKTRACE: perl(perl_run+0x4b8) [0x43ef78] STACKTRACE: perl(main+0x102) [0x419372] STACKTRACE: /usr/lib64/libc.so.6(+0x2814a) [0x7f5f1678014a] STACKTRACE: /usr/lib64/libc.so.6(__libc_start_main+0x8b) [0x7f5f167802 +0b] STACKTRACE: perl(_start+0x25) [0x4193b5]

    One needs to investigate the function names for e.g. +0x5542b of libcurl and 0x11d7b of Curl.o. Note that you will need to observe the make's output to find out how to recreate Curl.o as this is deleted at the end.

    And another alternative is to compile libcurl and this module with -g and run perl through gdb. You do not need to recompile perl with -g.

    Also, note that when compiling current version of this module it complains about a lot of deprecations.

    few hours Edit: The version of my libcurl is 8.2.1. According to the https://metacpan.org/dist/Net-Curl/changes, the latest (v0.56) Net::Curl::Multi has been adjusted for libcurl 8.7.1. So, it is worth retrying with exactly that libcurl version.

    It also fails with libcurl 8.9.1

    bw, bliako

      I've downloaded libcurl 8.7.1 and recompiled Net::Curl with it. It doesn't help, behavior is unchanged.

      Thanks anyway for the idea!