in reply to Removing duplicates in large files

You are using the wrong tool to solve your problem.

If you dont want to install MySQL, most any other DB will do (even Access probably). If the other DB doesn't allow uniqueness constraints on fields then just SELECT DISTINCT instead.

This might seem like alot of work, but if properly set up, you can use it over and over to process this file whenever you need to. Or you could just start using the DB instead anyway since it will make your life a lot easier anyway.

-stvn

Update: The more i re-read your post the more confused I am getting. Are these already in a database (RDMS)? Or is it in a text-file-database? If its already in a RDMS then just do a SELECT DISTINCT <column name> FROM <tablename> to get what you want, otherwise, see above. If you can't install the DB on the server, do it on your local machine and download the file. And for sure,.. don't do something like this with a CGI script, it will surely timeout no matter what you do.

Replies are listed 'Best First'.
Re: Re: Removing duplicates in large files
by Anonymous Monk on Jan 30, 2004 at 20:26 UTC
    Thank you very much that is pretty much what I thought. It is a text based file. I am using CGI since this is an internet based software pagage I am developing. So I have No choice but to use SQL, can't say I am that familiar with it. But I think it is time I get familiar because I have a lot of these type of files and CGI will never do the job.. Thanks again.