in reply to Getting/handling big files w/ perl

The first step in optimizing any perl script is to profile it to find out where it's spending most its time and focus on those parts before moving on. The best module that I'm aware of to do the profiling is Devel::NYTProf.

Replies are listed 'Best First'.
Re^2: Getting/handling big files w/ perl
by Gisel (Novice) on Nov 17, 2014 at 05:37 UTC

    I am a big booster of Devel::NYTProf, and have found some surprising and counter-intuitive results from it. However, in the present case my time is currently being spend outside of perl, in system calls to other executables that I do not have control of (and can’t profile inside of). I do use Timer::Simple and Time::HiRes to keep track of time spent on system calls.

    My lack of knowledge is along the lines of: are “wget (downloaded and compiled) and /usr/bin/gzip (part of distro) blunt tools for my needs”, or are they sharp tools via pedigree and refinement. My sense is that since they are designed to address a whole class of problems— vs. a perl module (or family of modules) designed to address a specific problem. Ergo, a more efficient perish solution is probably worth the effort. Unless I hear otherwise.

    Also, given the CPAN’s numerous code trees, (seemingly) providing different approaches, a nudge in the most productive direction would be appreciated.

    Thanx