Update: see Re^2: Debug background Perl procs started with compiled C# code (breaks only on AWS) for what I believe is a much simpler way to attempt to reproduce the issue. I can't say for sure what's happening, but the result is the same effect.

Hey fellow Monks,

I have a Perl issue, that's more an OS issue dealing with a Windows sub-cmd calling a Perl application while in the background. I'm hoping one of the Windows Monks might be able to help guide in how I can further troubleshoot (before I open an AWS ticket).

I can only repro the problem on an Amazon Web Services Windows Server. I can not reproduce the issue on a Windows box locally, or anywhere else. I believe the issue is being caused by one of the AWS default installed software applications.

I would greatly appreciate anyone who develops on Windows to test this out and provide feedback, and for those who can debug exe apps and can spare some time, I'll spin you up an AWS instance for testing (/msg, or steveb at cpan email).

Here is my process flow:

• perl script is run in the background (with Proc::Background). Script is a network listener, listening for incoming Perl module build/test requests to run against all locally installed Perl instances
• script qx's out a call to a C# application, which in turn runs a sub-proc in a new sub-cmd (hidden cmd.exe window) that calls a different Perl script (cpanm)
• in Task Manager, the sub-cmd opens, so does the Perl instance that is in control of cpanm, but this new perl sits still; zero CPU use

This works fine on *nix/Perlbrew, even on AWS systems. I believe it is because the sub-cmd controlling the perl cpanm run is held open by the OS (or other software), and just can't return (in fact, it doesn't appear to start cpanm).

The problem ONLY happens when running the tester on Amazon Web Services Windows servers.

My software works fine on all Windows systems other than AWS, and I have confirmed this repeatedly.

To repro:

Download and install Git for Windows: git, then:

mkdir c:\repos
cd c:\repos
git clone https://github.com/stevieb9/berrybrew
cd berrybrew
bin\berrybrew config # type 'y'
exit

# open new cmd window
berrybrew install 5.24.0_64
berrybrew switch 5.24.0_64
exit

# open new cmd window
cd c:\repos
cpanm Test::BrewBuild
bbtester start

# you can use your own CPAN module below, so long as
# it has a git repo; Test::BrewBuild will take care of
# the details. A small module repro's quicker

git clone https://github.com/stevieb9/file-edit-portable
cd file-edit-portable

bbdispatch -t localhost
[download]

It'll hang here. On a non-AWS system it doesn't.

NOTE: it works perfectly fine when running bbtester in the foreground (ie: bbtester --fg).

Although this condition can not be easily reproduced, I'll still show some code, as this after all, is PerlMonks.

    if ($^O =~ /MSWin/){
        $log->_6("on Windows, using work dir $work_dir");

        my $t;

        for (split /;/, $ENV{PATH}){
            if (-x "$_/perl.exe"){
                $perl = "$_/perl.exe";
                last;
            }
        }
        for (split /;/, $ENV{PATH}){
            if (-e "$_/bbtester"){
                $t = "$_/bbtester";
                last;
            }
        }
        $log->_6("using command: $perl $t --fg");

        @args = ($t, '--fg');
    }

...

    my $bg;

    if ($^O =~ /MSWin/){
        $bg = Proc::Background->new($perl, @args);
    }
    else {
        $bg = Proc::Background->new(@args);
    }
[download]

Full source of this part of the app can be found at here. Entire distribution, Test::BrewBuild.

Here's a snip of the C# berrybrew code. Process is a System.Diagnostic class.

internal static void Exec(StrawberryPerl perl, string command, string 
+path_env)
{
    Console.WriteLine("Perl-" + perl.Name + "\n==============");
 
    process = new Process();
    ProcessStartInfo startInfo = new ProcessStartInfo();
    startInfo.WindowStyle = ProcessWindowStyle.Hidden;
                                                            
    System.Environment.SetEnvironmentVariable(
        "PATH", 
        String.Join(";", perl.PerlPath, path_env)
    );

    startInfo.FileName = "cmd.exe";
    startInfo.Arguments = "/c " + perl.PerlPath + @"\" + command;
    process.StartInfo.RedirectStandardOutput = true;
    process.StartInfo.RedirectStandardError = true;
    process.StartInfo.UseShellExecute = false;
    process.Start();

    Console.WriteLine(process.StandardOutput.ReadToEnd());
    Console.WriteLine(process.StandardError.ReadToEnd());
    process.WaitForExit();
}
[download]

The rest of the code for the Windows berrybrew binary, here.

Again, I know this is a stretch, but if any Monks know how to debug Windows executables properly, I'd be willing to spin you up an AWS instance for troubleshooting.

-stevieb