in reply to Re^4: Search hex string in vary large binary file
in thread Search hex string in vary large binary file

In the OPs case, given he only wants a yay or nay answer ... parsing the entire file is a waste of cpu cycles, time, and effort.

You probably have much more experience with working with big data than I do, but in this case: On my machine, on a single 1.6GB video file where there is no match, your code takes 1.6s to complete (when working from disk cache), whereas a patched MP4::Info comes up with an answer in less than 0.1s. And MP4::Info scans 41GB of video files in under a second. Example code:

use MP4::Info 'get_mp4tag';
my $tag = get_mp4tag($ARGV[0]);
print $tag && $tag->{HDVD} && $tag->{HDVD}==2
    ? "Match! $ARGV[0]\n" : "No Match!\n";

[download]

In the general case, I tend to think the right tool for the job is much more likely to be a module (if it exists) - except maybe in the case of large amounts of input data, where optimizations may be necessary.

P.S. I'm sure you've read You can't parse (X)HTML with regex :-)

Replies are listed 'Best First'.
Re^6: Search hex string in vary large binary file
by BrowserUk (Patriarch) on Feb 07, 2015 at 20:55 UTC
    your code takes 1.6s to complete ... patched MP4::Info less than 0.1s.

    Hm. Sample code? The patch?

    P.S. I'm sure you've read You can't parse (X)HTML with regex :-)

    Tim Bray, one of the guys that put together the XML spec. does (and apparently prefers to); but that's by-the-by ....

    "Parse", in the sense of read-tokenise-build a structure that represents the entire document: I probably could, but it'd be more work than I'd take on. Especially when there are free modules that will do that for me.

    But if you want to extract a few values from within a jumble of text for which there is no parser, regex is the way to go.

    So, if I don't give a fig for the structure of the document, I treat it as a "jumble of text"; and get the job done.

    All I need is a unique anchor. And there *always* is one.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
[reply]

      Here's my test script, the patch to MP4::Info is as described earlier. The "scan" code hopefully does your solution justice. Note this particular script currently only tests for the presence of the HDVD flag, it doesn't look at whether it has a value of 1 (720p) or 2 (1080p) - but that shouldn't make any difference for this benchmark.

      Read more... (2 kB)
[reply]
[d/l]
        Note this particular script currently only tests for the presence of the HDVD flag, it doesn't look at whether it has a value of 1 (720p) or 2 (1080p)

        So, you're benchmarking a script that finds what the OP is looking for; against a script that doesn't; on files that don't contain it; using a "ready made solution", that you had to patch -- to look for 1/6th of the information required -- and concluding its faster.

        Yeah right!

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
[reply]

In Section Seekers of Perl Wisdom