If speed is important, prefer readdir over glob (3d run results):

use strict;
use warnings;
use feature 'say';
use Data::Dump 'dd';
use Time::HiRes 'time';
use File::Glob ':bsd_glob';
use Fcntl ':mode';
use File::stat;
use Win32::LongPath;

STDOUT-> autoflush;

my $dir = 'c:/program files';

{ 
    print "testing glob... ";
    my $t = time;
    my @to_do = ( $dir );
    my @result;
    
    while ( my $item = shift @to_do ) {
        my $stat = stat $item;
        next unless $stat;
        my $mode = $stat-> mode;
        if ( $mode & S_IFREG ) {
            push @result, [ $item, $stat-> size ];
        }
        elsif ( $mode & S_IFDIR ) {
            unshift @to_do, 
                grep { !m{ /\.{1,2}$ }x } 
                    bsd_glob( "$item/{.,}*" )
        }
    }
    printf "%d files, %.03f s\n", scalar( @result ), time - $t;
}
{
    print "testing readdir... ";
    my $t = time;
    my @to_do = ( $dir );
    my @result;
    
    while ( my $item = shift @to_do ) {
        my $stat = statL $item;
        next unless $stat;
        if ( $stat-> { mode } & S_IFREG ) {
            push @result, [ $item, $stat-> { size }];
        }
        elsif ( $stat-> { mode } & S_IFDIR ) {
            my $d =  Win32::LongPath-> new;
            $d-> opendirL( $item ) or next;
            unshift @to_do, 
                map { "$item/$_" } 
                    grep { !m{ ^\.{1,2}$ }x } 
                        $d-> readdirL;
        }
    }
    printf "%d files, %.03f s\n", scalar( @result ), time - $t;
}

__END__

testing glob... 18670 files, 3.492 s
testing readdir... 18670 files, 1.863 s
[download]

Sorry I've re-written your code completely, it was for investigation only. (One minor complaint may be that grep {} glob(), glob() looks like (grep {} glob()), glob() was intended, but this complaint is irrelevant to results). Also irrelevant (to speed) and maybe distracting are details which have happened in final script (which is not too DRY to begin with): bsd_glob, File::stat, no file tests as such, and, also, use of Win32::LongPath itself. The latter is slightly slower than opendir/readdir, and if trees are grown in controlled environment, not really necessary.

I suspect the explanation is glob performs stat on produced items (as File::Find does, if I'm not mistaken), it can't be so much slower because of strings manipulation only. BTW, I observe similar difference on Linux.

There's a cheat in that same number of files was neatly reported above -- but lists may not be the same, your result may have differing numbers, I get differing numbers for e.g. c:\users. I didn't investigate if it's access rights issues, or some specially treated magic directories on Windows, or links, etc. By that time I already discarded all error logging :). Maybe not important if trees are grown in data files land.

Thanks vr ! It has really helped me a lot!

… I've re-written your code completely ...

OK, Good!

1) I rewrote my FS_sweep based on your proposal

my $d =  Win32::LongPath-> new;
…
$d-> readdirL;
[download]

Doing stress tests using C:, I got a number of problems. The script sometimes works. But often it stucks, loops and was difficult to kill. Had to use the ctrl-alt-del/activity handler to stop it. I think that the memory is overwrite by  $d-> readdirL;. The largest directory I have found returned by  $d-> readdirL; has 38252 entries. MS File Explorer says: 38250 objekt

2) I rewrote my FS_sweep using while ( my $name = $dir->readdirL() )

Below follows a script which can be used to test this approach.

This works much better. But there are still problems. The script is sometimes stuck (no cpu time is used) or looping (cpu time is used, but nothing is happening). When logging the found file pathes to a file, the frequency of the problem seem to be lower. There is probably some type of timing problem in  readdirL. By accessing C:/Users the problem is rather frequent. Sweeping less complicated file structure as C: seem to be OK!

File path like <C:/Users/bo/Application Data/À> are sometimes returned by readdirL()!?

Here are some results

dir: C:   #dirs: 79923 #files: 305816 #nodes: 385739  1/s: 5749
dir: D:   #dirs: 7776 #files: 115255 #nodes: 123031  1/s: 14499
dir: Q:   #dirs: 907 #files: 16095 #nodes: 17002  1/s: 12374
dir: C:   #dirs: 67183 #depth: 13 #files: 259099 #nodes: 326282  1/s: 
+5558 (skipping 'C:/Users')
[download]

The in C:/Windows found number of files are 193 less than shown by the MS File Explorer and for directories 36 less.

In the documentation of Win32::LongPath there is in one of the examples

# recurse if dir
  if (($file ne '.') && (($stat->{attribs}
    & (FILE_ATTRIBUTE_DIRECTORY | FILE_ATTRIBUTE_REPARSE_POINT))
    == FILE_ATTRIBUTE_DIRECTORY)) {
    search_tree ($name);
    next;
  }
[download]

What does $stat->{mode} & S_IFDIR correspond to?

Here is my test script:

use strict;
use warnings;
use 5.010;

use Path::Tiny qw( path );
use Data::Dump qw(dump dd ddx);

use Win32::LongPath;
use File::stat;
use Fcntl ':mode';
use Benchmark qw(:all);
binmode STDOUT, ":utf8";
binmode STDERR, ":utf8";

my @dir_skip = (
    '$RECYCLE.BIN', 'System Volume Information', 'Config.Msi', 'C:/Use
+rs'
    #, 'C:/AMD', 'C:/hp'
);
my $dir_skip        = join '|', map { quotemeta } @dir_skip;
my $dir_skip_regexp = qr {$dir_skip$};

sub do_dir {
    my $dir_path = shift;
    my $sub_ref  = shift;    # callback

    my $dir = Win32::LongPath->new;
    unless ( $dir->opendirL($dir_path) ) {
        warn "!! unable to open $dir_path ($^E)";
        return;
    }
    my @dir_name;

    while ( my $name = $dir->readdirL() ) {

        if ( $name =~ m{ ^[.]{1,2}$ }x ) { next; }
        my $path = "$dir_path/$name";

        my $stat = lstatL($path);

        if ( !defined $stat ) {
            next if $^E =~ /Åtkomst nekad/;
            warn "!! SKIP $^E <$path>";
            next;
        }
        if ( $stat->{mode} & S_IFREG ) {    # normal file
            $sub_ref->( $path, $stat );     # call callback
        }
        elsif ( $stat->{mode} & S_IFDIR ) {    # dir
            push @dir_name, $name;
        }
        else {
            warn "!! ? $name";
        }
    }
    return \@dir_name;
}

{
    my @to_do;
    my $max_depth;

    sub td_to_txt { dump @to_do; }

    sub td_clear { @to_do = (); $max_depth = 0; }

    sub td_down {
        push @to_do, [];
        my $depth = @to_do;
        $max_depth = $depth if $depth > $max_depth;
    }

    sub max_depth { return $max_depth }

    sub td_add {
        my $name = shift;
        @to_do = [] unless (@to_do);
        push @{ $to_do[-1] }, $name;
    }

    sub td_add_aref {
        my $dir_aref = shift;
        push @{ $to_do[-1] }, @$dir_aref;
    }

    sub td_path_next {
        return join '/', map { $_->[0] } @to_do;
    }

    sub td_remove_dir {
        my $aref    = $to_do[-1];
        my $removed = shift @$aref;    # remove dir
        return if @$aref;              # more dirs

        while ( $aref = $to_do[-1] ) {
            if ( !@$aref ) {
                $removed = pop @to_do;    # remove level
                next;
            }
            $removed = shift @{ $to_do[-1] };    # remove dir
            return if @$aref;                    # more dirs
        }
    }
}

sub FS_sweep {
    my $dir_path = shift;
    my $sub_ref  = shift;
    td_clear;
    td_down;
    td_add($dir_path);
    my $dir_cnt = 0;
    my $t0      = Benchmark->new;

    while ( my $dir_path = td_path_next ) {
        if ( $dir_path =~ m{$dir_skip_regexp} ) {
            warn "SKIPING DIR $dir_path";
            td_remove_dir;
            next;
        }

        $dir_cnt++;
        my $dir_name_aref = do_dir( $dir_path, $sub_ref );

        my $sub_dir_nof = @$dir_name_aref;
        if ( $sub_dir_nof > 1000 ) {
            warn "!! MANY SUBDIR  $sub_dir_nof in $dir_path";
        }

        if (@$dir_name_aref) {    # subdir
            td_down;
            td_add_aref($dir_name_aref);
        }
        else {
            warn '!! ! defined $dir_name_aref' if !defined $dir_name_a
+ref;
            td_remove_dir;
        }
    }
    my $td = timediff( Benchmark->new, $t0 );
    return $dir_path, $dir_cnt, max_depth, $td;
}

my @output;
my $file_cnt = 0;

sub file_log {
    my $file_path = shift;
    $file_cnt++;
    warn "!#      $file_cnt $file_path\n" if not $file_cnt % 10000;
}

sub summary {
    my $dir_path        = shift;
    my $dir_cnt         = shift;
    my $max_depth       = shift;
    my $td              = shift;
    my $node_cnt        = $dir_cnt + $file_cnt;
    my $node_per_second = $td->cpu_p > 0 ? $node_cnt / $td->cpu_p : -1
+;
    my $txt =
      sprintf
"\n\n!! FS_sweep summary dir: %s\n   #dirs: %d #depth: %d #files: %d #
+nodes: %d  1/s: %d\n",
      $dir_path, $dir_cnt, $max_depth, $file_cnt, $node_cnt, $node_per
+_second;
    $file_cnt = 0;
    return $txt;
}

my $ls_log = 1;    # activate listing of files in 'ls_log.txt'
my $log_fh;
$log_fh = path('ls_log.txt')->openw_utf8 if $ls_log;

sub FS_file_big {
    my $file_path     = shift;
    my $stat_hash_ref = shift;
    file_log($file_path);
    say {$log_fh} $file_path if $ls_log;
    my $size = $stat_hash_ref->{size};
    push @output, "BIG $file_path size: $size\n" if $size > 100000000;
}

sub output {
    if (@output) {
        say "Output:";
        say map { "$_\n" }
          grep { defined } @output[ 0 .. 100, 1000 .. 1010, 2000 .. 20
+10 ];
        say "END Output\n";
        @output = ();
    }
    STDOUT->flush;
}

say summary( FS_sweep( 'C:/Windows', \&FS_file_big ) );
output;

say summary( FS_sweep( 'C:', \&FS_file_big ) );
output;

foreach my $dev (qw{ }) { # add C D ...
    warn "!! START $dev: =======================================\n";
    say summary( FS_sweep( "$dev:", \&FS_file_big ) );
    output;
}
[download]

Hi, glad to hear your script was improved. To your questions:

(1) Re: S_IFDIR. Easy to check: as I see there are 2 entries/directories in my C:/Users tree with FILE_ATTRIBUTE_REPARSE_POINT bit set; for both S_IFDIR is also set. So, to strictly follow Win32::LongPath documentation as to what to treat as "directory" i.e. exclude such entries, -- use fragment you quoted from documentation. Stress tests are good, but my impression was your tree to monitor should be known beforehand, whether it can or cannot contain reparse points. And, I meant the same thing (about "controlled environment"), when I said built-in readdir might be enough vs. readdirL, if a tree is known to be grown by native speakers/users of single (system) Windows code page. For versatility, sure, Win32::LongPath should be preferred.

(2) Re: <C:/Users/bo/Application Data/À> - I don't understand (was this a question/problem?)

About freezes: that's not good. I can repeatedly sweep C:/Users and C:/Windows without issues, both with my and your scripts. At first, I suspected iterative readdirL (furthermore interspersed with lstatL calls) might be the reason, but, no, your script runs OK here. Still, to debug, maybe try to switch from iterative use to list context call. BTW, I don't observe any noticeable speed difference.

Lastly, about different total results of (various) Perl techniques and/or what Explorer reports: if you really want to pursue to the core, there's dichotomy with extensive logging, but of course you know the method.

Questions

- Which type of files and directories is the globe function returning?

- How can I improve my sweep function?
[download]

The OP already knows about File::Find... and it doesn't answer either of his questions.

Please. If you don't have anything useful to contribute, just... stfu.

The following is a JavaScript program I wrote which sweeps a directory tree and creates an unicode text file which contains all the file names in a directory. Runs under Windows XP/7/8/10. It should be saved with a .JS extension.

Some tasks are easier in perl, while others are easier in JavaScript. And although JavaScript does not require it, you can use $ signs in the variables, which I did, to make it more perlish. Lol

// This JavaScript program reads directory contents
// and saves the sorted list in a file in Unicode format.

$PATH = "C:\\WINDOWS";
$RECURSIVE = 0;
$OUTPUT_FILE = "Z:\\Output.txt";
$SAVE_DIR_SIZE = 0;
$SAVE_FULLPATH = 0;

// PROGRAM STARTS HERE:
$T = (new Date()).getTime();   // Get current time in milliseconds
$PATH_LENGTH = $PATH.length + (($PATH.slice(-1) == '\\') ? 0 : 1);
try { $FSO = new ActiveXObject('Scripting.FileSystemObject'); } catch(
+e) { Quit(1, 'Cannot access File System.'); }
if ($PATH.charAt(1) == ':')
{
  $DRIVE = $PATH.substr(0, 2);
  if (!$FSO.DriveExists($DRIVE)) Quit(2, 'Drive does not exist - ' + $
+PATH);
  if (!($FSO.GetDrive($DRIVE)).IsReady) Quit(3, 'Drive is not ready - 
+' + $PATH);
}
OUTPUT = [];              // create an array for file data
DIR($PATH);               // sweep directory and collect data
OUTPUT = OUTPUT.sort();   // sort all files by size (this line can be 
+removed)
OUTPUT.unshift($PATH);    // Save PATH starting point
OUTPUT.unshift($T);       // The first line of the output file will co
+ntain the exact time when this program was executed.

try                       // Write file...
{
  // Here we try to open the text file for writing using the CreateTex
+tFile() method.
  // The first "true" argument states that the output file will be ove
+rwritten.
  // The second "true" argument states that the output text file will 
+be in Unicode format.
  var $F = $FSO.CreateTextFile($OUTPUT_FILE, true, true);
  $F.Write(OUTPUT.join("\r\n"));  // Write contents of the array into 
+the file.
  $F.Close();
}
catch (e) { Quit(4, 'Cannot save file - ' + $OUTPUT_FILE); }
Quit(0, "SUCCESS!!!!!\n\nDIRECTORY CONTENTS OF\n\n" + $PATH + "\n\nSAV
+ED SUCCESSFULLY TO:\n\n" + $OUTPUT_FILE);


// This function reads the contents of one directory and saves the
// contents in the OUTPUT array.
function DIR($path)
{
  var $F = $FSO.GetFolder($path), $FC, $File, $FullName;

  // First we record all the sub-directories.
  for ($FC = new Enumerator($F.SubFolders); !$FC.atEnd(); $FC.moveNext
+())
  {
    $FullName = $FC.item();
    $File = $FSO.GetFolder($FullName);
    OUTPUT.push('+' + ($SAVE_DIR_SIZE ? '00000000000 ' : '') + DateOf(
+$File) + AttributesOf($File) + ($SAVE_FULLPATH ? $FullName : Shorten(
+$FullName))); if ($RECURSIVE) DIR($FullName);
  }

  // Then we record all the files.
  for ($FC = new Enumerator($F.files); !$FC.atEnd(); $FC.moveNext())
  {
    $FullName = $FC.item();
    $File = $FSO.GetFile($FullName);
    OUTPUT.push('-' + SizeOf($File) + DateOf($File) + AttributesOf($Fi
+le) + ($SAVE_FULLPATH ? $FullName : Shorten($FullName)));
  }
}

// This function returns the last modified date of a file or directory
+.
function DateOf($f) { return ('0000000000' + ($f.DateLastModified * 1)
+).slice(-13).substr(0, 10) + ' '; }

// This function returns the size of a file.
function SizeOf($f) { return ('00000000000' + $f.Size + ' ').slice(-12
+); }

// This function returns the file attributes in a nice formatted way.
function AttributesOf($f) { var $A = $f.Attributes;
return ($A & 1 ? 'R' : '-') // Read-only
  + ($A & 2 ? 'H' : '-')  // Hidden file
  + ($A & 4 ? 'S' : '-')  // System file
  + ($A & 8 ? 'V' : '-')  // Volume label (attribute is read-only)
  + ($A & 16 ? 'D' : '-')  // Directory
  + ($A & 32 ? 'A' : '-')  // Archive
  + ($A & 1024 ? 'L' : '-')  // Link or shortcut file
  + ($A & 2048 ? 'C' : '-') + ' ';  // Compressed file
}

// Removes the first part of a fullpath.
function Shorten($n) { return ($n + '').slice($PATH_LENGTH); }

// Terminates the program and maybe displays a message.
function Quit($errorcode, $msg) { if (typeof($msg) == 'string') WScrip
+t.Echo($msg); WScript.Quit($errorcode); }
[download]

Here's a sample output produced by this program:

1593733100343
C:\WINDOWS
+1578861460  ----D--- Cursors
+1578861460  ----D--- Driver Cache
+1578861460  ----D--- Help
+1578861460  ----D--- Media
+1578861460  ----D--- NLDRV
+1578861460  ----D--- Provisioning
+1578861460  ----D--- Resources
+1578861460  ----D--- WinSxS
+1578861460  ----D--- ehome
+1578861460  ----D--- msagent
+1578861460  ----D--- mui
+1578861460  ----D--- pchealth
+1578861460  ----D--- repair
+1578861460  ----D--- security
+1578861460  ----D--- system
+1578861460  ----D--- system32
+1578861460  ----D--- twain_32
+1578861460  -H--D--- inf
+1578861460  R-S-D--- Fonts
+1578861772  -HS-D--- Installer
+1578862010  ----D--- Registration
+1578862048  ----D--- srchasst
+1578862072  R---D--- Web
+1578862902  ----D--- SoftwareDistribution
+1578863966  ----D--- SHELLNEW
+1578868700  ----D--- RegisteredPackages
+1578869970  -H--D--- $NtUninstallKB888111WXPSP2$
+1578888466  -H--D--- $MSI31Uninstall_KB893803v2$
+1578899930  ----D--- Microsoft.NET
+1578899952  R-S-D--- assembly
+1579014252  -HS-D--- Recycled
+1579107350  -H--D--- $NtUninstallWIC$
+1579107390  -H--D--- $NtUninstallKB894476$
+1579107394  -H--D--- $NtUninstallKB909394$
+1579150942  -H--D--- $NtUninstallWdf01009$
+1579150946  -H--D--- $NtUninstallwinusb0200$
+1579151832  -H--D--- $NtUninstallKB942288-v3$
+1591766356  -H--D--- PIF
+1593196520  ----D--- apppatch
+1593196520  ----D--- ime
+1593196520  ----D--- peernet
+1593227820  ----D--- temp
00000000000 1578898458  -----A-- ativpsrm.bin
00000000000 1593196522  -----A-- Sti_Trace.log
00000000036 1578862012  -----A-- vb.ini
00000000037 1578862012  -----A-- vbaddin.ini
00000000038 1271437200  -----A-- avisplitter.ini
00000000042 1580688388  -----A-- boxworld.ini
00000000048 1593666958  -----A-- wiaservc.log
00000000080 1141128000  -----A-- explorer.scf
00000000109 1580687782  -----A-- WCHESS.INI
00000000116 1592963652  -----A-- NeroDigital.ini
00000000140 1578896578  -----A-- gife.ini
00000000159 1593729422  -----A-- wiadebug.log
00000000203 1593625384  -----A-- GIBW.INI
00000000227 1578871656  -----A-- system.ini
00000000335 1580305028  -----A-- nsreg.dat
00000000376 1578864048  -----A-- ODBC.INI
00000000383 1578893046  -----A-- SNAP.INI
00000000706 1578898910  -----A-- SS_SLIDE.INI
00000000749 1578862066  RH---A-- WindowsShell.Manifest
00000000808 1593521706  -----A-- win.ini
00000001272 1141128000  -----A-- Blue Lace 16.bmp
00000001405 1141128000  -----A-- msdfmap.ini
00000002048 1593729420  --S--A-- bootstat.dat
00000003626 1593666956  -----A-- WindowsUpdate.log
00000004079 1593229038  -----A-- IF40LE.INI
00000004161 1578862110  -----A-- ODBCINST.INI
00000009522 1141128000  -----A-- Zapotec.bmp
00000010752 1141128000  -----A-- hh.exe
00000015360 1141128000  -----A-- TASKMAN.EXE
00000016730 1141128000  -----A-- FeatherTexture.bmp
00000017062 1141128000  -----A-- Coffee Bean.bmp
00000017336 1141128000  -----A-- Gone Fishing.bmp
00000017362 1141128000  -----A-- Rhododendron.bmp
00000018944 1141128000  -----A-- vmmreg32.dll
00000025600 1141128000  -----A-- twunk_32.exe
00000026582 1141128000  -----A-- Greenstone.bmp
00000026680 1141128000  -----A-- River Sumida.bmp
00000035363 1312180208  -----A-- atiogl.xml
00000038065 1075075200  -----A-- EPHEM12.HLP
00000048680 1141128000  -HS----- winnt.bmp
00000048680 1141128000  -HS----- winnt256.bmp
00000049680 1141128000  -----A-- twunk_16.exe
00000050688 1141128000  -----A-- twain_32.dll
00000059392 1463460046  -----A-- STARFIELD.SCR
00000062976 1141128000  -----A-- SPGRMR.DLL
00000064512 1463460042  -----A-- CYLFRAC.SCR
00000065832 1141128000  -----A-- Santa Fe Stucco.bmp
00000065954 1141128000  -----A-- Prairie Wind.bmp
00000065978 1141128000  -----A-- Soap Bubbles.bmp
00000069120 1141128000  -----A-- NOTEPAD.EXE
00000069632 1115163808  -----A-- ALCMTR.EXE
00000075264 1463459448  -----A-- 3DTEXT.SCR
00000086016 1153516476  -----A-- SOUNDMAN.EXE
00000094784 1141128000  -----A-- twain.dll
00000128000 1463460046  -----A-- MATRIX.SCR
00000130048 1141128000  -----A-- SOFTKBD.DLL
00000146432 1141128000  -----A-- regedit.exe
00000149504 1463460046  -----A-- SSSTARS.SCR
00000220160 1141128000  -----A-- MSCANDUI.DLL
00000223744 1463460044  -----A-- BUTTERFLIES.SCR
00000250880 1141128000  -----A-- SPTIP.DLL
00000256192 1141128000  -----A-- winhelp.exe
00000260640 1075075200  -----A-- EPHEM12.SCR
00000282624 1118694920  -----A-- SLIDESHOW.SCR
00000283648 1141128000  -----A-- winhlp32.exe
00000316640 1578868716  -----A-- WMSysPr9.prx
00000626397 1593382024  -----A-- setupapi.log
00000720896 1579922638  -----A-- iun6002.exe
00001032192 1141128000  -----A-- explorer.exe
00001183744 1159470048  -----A-- RtlUpd.exe
00002157568 1160606578  -----A-- MicCal.exe
00002808832 1146777996  -----A-- ALCWZRD.EXE
00002879488 1147820666  -----A-- SkyTel.exe
00009709568 1146778514  -----A-- RTLCPL.EXE
00016269312 1162259394  -----A-- RTHDCPL.EXE