in reply to transposing and matching large datasets

If the record lines in the files are fixed column width you may be able to use DBD::AnyData with DBI.


DWIM is Perl's answer to Gödel
  • Comment on Re: transposing and matching large datasets

Replies are listed 'Best First'.
Re^2: transposing and matching large datasets
by wannabemonk (Initiate) on Aug 08, 2007 at 21:00 UTC
    Ok I will look into it. Thank you.
Re^2: transposing and matching large datasets
by moritz (Cardinal) on Aug 09, 2007 at 07:10 UTC

    If they are not fixed column width I'd still recommend to use a form of database, because this kind of problem is what databases are good at.

      because this kind of problem is what databases are good at.

      If the data is already in the database maybe--but only maybe. It depends on whether it is indexed correctly for this particular operation.

      If the data is not in a database, then in the time you spend writing the script to load the data into the database, the operation is completed using simple flat file operations. And that's before you actually load the data, index it, and then perform the join and export all the data back to a flat file.

      If there is an ongoing need for relational operations upon the dataset, then the costs of importing there may be amortisable over the long term. But to import the data to a db, just to join it and export it all again is a complete nonsense.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.