How do the caller tell if S returned a scalar or an array?

Perl subs can only return a list of scalars. They cannot return arrays.

So you're asking how to determine if the value returned by S originated in an array or not. That's insane. What problem are you actually trying to solve?

I am building a transport system, I'm going to call user funcions, collecting their output, serialize it with JSON and transmit through a socket.

I don't know what will be in those functions, but would like to serialize to as correct JSON as possible.

The posted code shows that perl it self somehow were able to tell the differense, I would like to do the same in my program.

The posted code shows that perl it self somehow were able to tell the differense...

It doesn't. Apart from the value of $mode, these two lines do the same thing.

    return 42     if $mode eq 'a';
    return (42)   if $mode eq 'b';
[download]

The parentheses are superfluous; they do not change the behavior in this case.

In the final case:

    my @l = (42);
    return @l     if $mode eq 'c';
[download]

The return value depends on the context in which the caller of this function invoked it. When a function called in scalar context (as you've shown here) returns an array, that return value evaluates to the number of elements in the array.

Oh, and the parentheses in the assignment to the array are also superfluous. This code is equivalent:

my @l = 42;

If you'd like to learn more about context, the first chapter of Modern Perl explains it in detail.

I don't know what will be in those functions, but would like to serialize to as correct JSON as possible.

Functions return lists of scalars. It is as correct as possible to transmit a list of scalar. You could use a JSON array for that.

$json->encode( [ f() ] )
[download]

The equivalent of

my $s = f();
my @a = f();
[download]

is

my $s = $json_decode(...)->[0];
my @a = @{ $json_decode(...) };
[download]

How do the caller tell if S returned a scalar or an array?

The answer is, it's actually not up to S. If S is called in scalar context, a scalar is returned. If S is called in list context, a list is returned. If S is called in void context (a special case of scalar context), nothing is returned.

use strict;  
use warnings; 

sub mysuba {
    my @fred = qw(The quick brown fox);
    return @fred
}

sub mysubb {
    return qw(The quick brown fox)
}

my $resa = mysuba();
print "$resa\n";

my $resb = mysubb();
print "$resb\n";
[download]

4
fox
[download]

Well, if what the function returns depends on $run, you know ahead of time what mode it is in

This is really easy to avoid by documenting what the function returns, and always returning the same thing regardless of context