Clear questions and runnable code get the best and fastest answer |
|
PerlMonks |
Re: Best way to store/access large dataset?by erix (Prior) |
on Jun 22, 2018 at 16:14 UTC ( [id://1217220]=note: print w/replies, xml ) | Need Help?? |
You still haven't really clearly said what the (filtering) query would look like. So after enlarging your attributes.txt to 1 million rows I used postgres as a front-end to read that whole file (via file_fdw). To read and "calculate" the whole file takes 2 seconds. This is the SQL I used:
A more interesting part would probably be a WHERE-clause, or possibly an ORDER BY clause, that you would need but I don't know how that would look from what you've said so far. UPDATE: I typoed the order of the column names so fixed that.
In Section
Seekers of Perl Wisdom
|
|