in reply to Checking LInes in Text File

For small files read the file a line at a time and add the lines to a hash.

When you have read a line first check that it's not in the hash already. If not, print it to the output file and insert it into the hash, otherwise skip to the next line.

If you can't figure that out, write some code that is your best guess at how to do it, ask again and include your code.

DWIM is Perl's answer to Gödel

Replies are listed 'Best First'.
Re^2: Checking LInes in Text File
by Anonymous Monk on Jun 01, 2006 at 20:47 UTC
    Grandfather, I have a further question related to your suggestion. If I have hundres of lines of data like this:

    tag1:xxxxxxx tag2:xxxxxx tag3:xxxxxxx tag4:yyyyy

    How can I remove all lines that have the same tags 1 through 3 and replace it with a single line that has a new tag4? Currently I am able to remove all excess lines with the same tags 1 through 3 using your method but am unable to change tag4 because the hash method works by not writing subsequent values. Hence once I find out I have a duplicate it is too late to change it as the first has already been written. Any suggestions? Thanks!
[reply]

      Provide half a dozen lines of sample data, the test code you are currently using, and a sample of the output you expect to see.

      For the test code it is easiest to use a __DATA__ section for the test data rather than an external file and simply print the result rather than generating an external file.

      DWIM is Perl's answer to Gödel
[reply]
        Thanks GrandFather!

        My data would looks like this:
        MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
        MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
        MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
        MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0
        MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
        MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
        MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
        MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA1
        so every set of data (MCAT, PCAT, LMAT) is unique except for the fourth and eight which differ only by TYPE.

        I would like to roll the fourth and eight together so it would look like this:

        MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
        MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
        MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
        MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0, KA1
        MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
        MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
        MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1

        This leaves me with a total of 7 data groups instead of 8.

        So far my code doesn't even come close to accomplishing doing this....
[reply]

In Section Seekers of Perl Wisdom