in reply to OT: routine for finding duplicate entries from a MySQL Table

You need some SQL here. The following query will show you which questions have a duplicate value.

SELECT 
    questions, 
    CONCAT(option_a, option_b, option_c, option_d) AS options, 
    COUNT(*) AS howmany
FROM 
    yourtable
GROUP BY
    questions, options
HAVING
    howmany > 1

[download]

But if you simply want to delete the duplicates, then use ALTER TABLE to add a unique index to your table. Notice that you may need to qualify your columns with the number of characters to include. (See the appropriate manual page for details.) 

ALTER IGNORE TABLE yourtable 
    ADD UNIQUE KEY (questions, option_a, option_b, option_c, option_d)
+;

[download]

This will also prevent duplicates to happen in the future.

Replies are listed 'Best First'.
Re: Re: OT: routine for finding duplicate entries from a MySQL Table
by CountZero (Bishop) on Feb 04, 2004 at 13:53 UTC
    The idea to add a UNIQUE index is OK, but the way you propose it is not correct.

    You are adding a UNIQUE KEY which consists of both the questions field and all the option_? fields, which means that for the same question but with a different set of options, you get a different entry in the index and hence the "NOT UNIQUE" error doesn't trigger.

    The solution is therefore to add a UNIQUE KEY on the questions field only and then you can add only one record with this question and all later records with the same question (but with perhaps different options) will not get added to the database.

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[reply]
[d/l]
[select]

      Well, the problem is that the OP wasn't very clear about what is a duplicate record. He mentioned an ID, but it isn't in the table description, so I assumed that a primary key existed, apart the columns that were mentioned.

      Anyway, if the problem is (more trivially) to find out if several records have the same "questions" column, then it's even simpler:

      SELECT 
    questions,
    COUNT(*) AS howmany
FROM 
    yourtable
GROUP BY
    questions
HAVING
    howmany > 1

      [download]

      To remove such duplicates, on the same vein:

      ALTER IGNORE TABLE yourtable ADD UNIQUE KEY (questions)

      [download]
[reply]
[d/l]
[select]
        Sorry for the vague description of the problem above. There's actually an AUTO_INCREMENT PRIMARY KEY ques_id that exists in the table, and many other columns that I did not show.

        Your suggestion is highly appreciated. Thanks.
[reply]

In Section Seekers of Perl Wisdom