"Merge" ... "datasets" ... "by (something)". Gee, that sounds like SQL. 

  select id, name from fileone where id in (select id from filetwo)

[download]
Unfortunately, I'm not having a lot of luck getting SQL::Statement (which underpins DBD::CSV) into accepting this. Could be something wrong with my syntax, but this doesn't seem that convoluted yet. SQL ERROR: Bad table or column name 'select id from f2' has chars not alphanumeric or underscore! (?) The full code I'm trying is: 
#! /usr/bin/perl
use strict;
use warnings;

use DBI;

my $dbh = DBI->connect("DBI:CSV:f_dir=.;csv_eol=\n;csv_sep_char=\t");
unlink 'out';
$dbh->do("create table out (id integer, name char(64))");
$dbh->{csv_tables}{f1}{col_names} = [qw(id name)];
$dbh->{csv_tables}{f2}{col_names} = [qw(id)];

my $d = $dbh->selectall_arrayref('select id, name from f1 where id in 
+( select id from f2 )', {});

[download]
and something like this should work (assuming tab separators), but, like I said, I'm having some minor problems with it. Assuming we get the kinks worked out (could be my SQL, could be SQL::*), I would think this to be the simplest situation. And, if it's a problem with SQL::*, then moving your "huge" data to a real database may be wise anyway, and that'd solve the problem right out as their SQL parser/engine would probably handle it ;-)

In reply to Re: Merge two huge datasets by ID by Tanktalus
in thread Merge two huge datasets by ID by dpangpang

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.