Make a file handle hot

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks! I have a script where I'm writing a lot of files. It slows my computer up quite a bit and I've done some reading on it and it looks like making the file handle hot might be the answer - as far as freeing up memory and speeding things up. I can see that $| looks to be the key. Any help on modifying the code below would to make the file handle hot so I'm not killing my computer would be greatly appreciated

my $calls_dir3 = "Bing/1Parsed/Html2/";
    opendir( my $search_dir2, $calls_dir3 ) or die "$!\n";
    my @files = grep /\.txt$/i, readdir $search_dir2;
    closedir $search_dir2;
    print "Got ", scalar @files, " files\n";

    # proxies
    open my $fh2, '<', 'proxies.txt' or die $!;
    chomp( my @proxies = <$fh2> );
    close $fh2;

    foreach my $file (@files) {
        my %seen         = ();
        my $current_file = $calls_dir3 . $file;
        my $proxy        = shift @proxies;
        $proxy;
        print "Current proxy:$proxy\n";
        open my $FILE, '<', $current_file or die "$file: $!\n";


        my $x=1;
        make_path('Bing/1Parsed/Html3/');
        while ( my $row = <$FILE> ) {
            open my $fh1, ">", "Bing/1Parsed/Html3/$file.$x.html"
              or die("Could not open file. $!");

            #my $rnumber2  = rand(1999);
            $x = $x + 1;
            chomp $row;
            print "$row\n";
            my $xml1 = $row;

            $fh1->print ("<meta name=" . chr(34) . "keywords" . chr(34
+) . "content=" . chr(34) . ($row) . chr(34) .">");

            # create useragent
            my $ua = LWP::UserAgent->new;
            $ua->agent('Mozilla/5.0');
            $ua->timeout('180');

            # Use this UA/Proxy to fetch something....
            $ua->proxy( ['http'], 'http://' . $proxy );

            my $xml2 = get $xml1;
            $xml2;
            
            #add pause
            #my ( $x, $y ) = ( 1, 2 );
            #my $result = int( rand( $y - $x + 1 ) ) + $x;

            #print "pausing ";
            #print $result;
            #print "seconds";
            #sleep ($result);

            $fh1->print("\n");

            print $xml2;
            $fh1->print($xml2);

            close $fh1;
            $xml2 = 1;
        }
    }
[download]

Comment on Make a file handle hot Download Code

Replies are listed 'Best First'.
Re: Make a file handle hot by choroba (Cardinal) on Mar 08, 2016 at 14:52 UTC
You can try `$fh1->autoflush(1);` [download] after opening it, but I doubt it would speed up the process. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re: Make a file handle hot by neilwatson (Priest) on Mar 08, 2016 at 15:01 UTC
You might test using Benchmark Neil Watson watson-wilson.ca	[reply]
Re: Make a file handle hot by kennethk (Abbot) on Mar 08, 2016 at 19:56 UTC
Both previous answers have value. Even better than Benchmarking would be profiling. I use Devel::NYTProf, but I vaguely recall there's a hotter, potentially better tool. You should read the documentation in perlvar about $\|. That and select should help with understanding hot pipes. I would be very surprised if setting an output channel to autoflush improves your performance. The whole point of output buffering is to improve performance, which means you would be undermining an existing optimization. #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.	[reply]
Re: Make a file handle hot by perlfan (Parson) on Mar 09, 2016 at 04:11 UTC
I'd add some forking to the mix, let the OS schedule the I/O and network access for you. Experiment to see how many children you need to saturate your I/O and networking.	[reply]