in reply to processing large files

As a network analyst for a tier-1 ISP, I can happily report that Perl can, and on a daily basis, does, process files in the neighborhood of several hundred gig. Are you sure that your limitation is not hardware-related?

Replies are listed 'Best First'.
Re: Re: processing large files
by Anonymous Monk on Jul 07, 2001 at 01:12 UTC
    How could it be hardware related if these files are on
    disk at that size and C++ programs process them?
    how do you process your files with perl? Is it a specfic
    perl? or compile option? or algorithm?
    thanks

    -E
      Easily! However, you never said that you had other programs that were successfully processing your data. =) As far as my processing is concerned, I churn through Cisco NetFlow data, along with bgp tables, route summaries, snmp data, merge it all together and spit it back out again to create traffice matrices. The algorithms aren't that complicated at all; it's just a shite-load of data that I process continuously. If I tried to run on anything less than the E450 that it runs on, I would be in trouble.