data extraction from badly formated txt files

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I'm writing a script to extract data from a lot of txt files. Here is how each txt file is formatted:

 18884 alfCet  2.5 1.6 1.4 1U  43.3 241.9   22903.0    151.3     747.0
 18925 gamPer  2.9 0.7 0.6     54.6 319.3   14474.0    120.3     394.0
 19058  roPer  3.4 1.6 1.8 2U  57.5 293.4   22005.0    148.3    1122.0
 19275         4.9 0.0 0.1  C  43.7 346.9   18311.0    135.3    1700.0
 19356 betPer  2.1-0.1 0.0 3P  58.0 297.4   20842.0    144.4     417.0
 19373 iotPer  4.0 0.6 0.5  C  56.5 313.2   16303.0    127.7     510.0
 19476 kapPer  3.8 1.0 0.7  C  57.7 304.7   18094.0    134.5     707.0
 19656 omgPer  4.6 1.1 0.8     58.6 294.9   23508.0    153.3    1330.0
 19787 delAri  4.4 1.0 0.8     54.1 259.4   30779.0    175.4     524.0
 20150 zetAri  4.9 0.0 0.1  C  55.4 260.9   33579.0    183.2     638.0
 20336         4.8-0.1 0.0 1U  50.0 338.0   13931.0    118.0     453.0
 20468         4.8 1.5 1.1     59.9 284.2   22993.0    151.6    2761.0
 20630 kapCet  4.8 0.7 0.6 1P  45.9 237.1   24642.0    157.0     463.0
 20644         4.5 1.5 1.2 1M  59.4 274.0   32184.0    179.4     729.0
 20677    Per  5.0 0.0 0.1  C  60.1 302.5   20469.0    143.1     392.0
 20902 alfPer  1.8 0.5 0.5 1M  58.8 315.2   16320.0    127.7     430.0
 21120 omiTau  3.6 0.9 0.7  C  50.7 241.9   36123.0    190.1    3694.0
 21278         5.0-0.1 0.0  C  59.6 314.2   16296.0    127.7     464.0
 21291         4.2 0.4 0.4 1P  54.4 331.8   14214.0    119.2     373.0
 21364  xiTau  3.8-0.1 0.0 2U  51.6 242.2       0.0      0.0       0.0
 21389         4.5 0.6 0.5 1U  55.1 330.4   13585.0    116.6     386.0
 21428    Per  4.7-0.1 0.0  D  59.7 315.1       0.0      0.0       0.0
[download]

Here is my code so far:

@filecheck = <*.txt>;

    foreach $file (@filecheck) {
        open FILE, "$file" or die "\nError reading file";
        @lines = <FILE>;
        close FILE;

        open FILE, "$file" or die "\nError reading file";
        foreach $line (@lines) {
            @words = split ' ', $line;
            if (@words [1] eq $star) {
                print "\n@words[var]";
            }
        }
    }
[download]

As you can see, I break the txt data into arrays via spaces. Here is why the formatting of the file matters. Sometimes, there are blank entries where data should be or a negative sign inbetween two numbers that should be considered seperate. This makes the array of the words smaller than it should be. Since the data that is to be outputed depends on its position in the array, the wrong data is outputed.

Is there an easier way to do this or break the txt file down to make this possible?
Thanx.

Comment on data extraction from badly formated txt files Select or Download Code

Replies are listed 'Best First'.
Re: data extraction from badly formated txt files by thelenm (Vicar) on Oct 24, 2002 at 19:46 UTC
I wouldn't say it's badly formatted, it's just formatted in fixed-width columns instead of whitespace-separated. Try something along the lines of this solution, using unpack: #!/usr/bin/perl use warnings; use strict; while (<DATA>) { my @words = unpack "A6A7A5A4A4A3A6A6A10A9A10", $_; s/^\s+// for @words; # Now @words contains your data print "@words\n"; } __DATA__ 18884 alfCet 2.5 1.6 1.4 1U 43.3 241.9 22903.0 151.3 747.0 18925 gamPer 2.9 0.7 0.6 54.6 319.3 14474.0 120.3 394.0 19058 roPer 3.4 1.6 1.8 2U 57.5 293.4 22005.0 148.3 1122.0 19275 4.9 0.0 0.1 C 43.7 346.9 18311.0 135.3 1700.0 19356 betPer 2.1-0.1 0.0 3P 58.0 297.4 20842.0 144.4 417.0 19373 iotPer 4.0 0.6 0.5 C 56.5 313.2 16303.0 127.7 510.0 19476 kapPer 3.8 1.0 0.7 C 57.7 304.7 18094.0 134.5 707.0 19656 omgPer 4.6 1.1 0.8 58.6 294.9 23508.0 153.3 1330.0 19787 delAri 4.4 1.0 0.8 54.1 259.4 30779.0 175.4 524.0 20150 zetAri 4.9 0.0 0.1 C 55.4 260.9 33579.0 183.2 638.0 20336 4.8-0.1 0.0 1U 50.0 338.0 13931.0 118.0 453.0 20468 4.8 1.5 1.1 59.9 284.2 22993.0 151.6 2761.0 20630 kapCet 4.8 0.7 0.6 1P 45.9 237.1 24642.0 157.0 463.0 20644 4.5 1.5 1.2 1M 59.4 274.0 32184.0 179.4 729.0 20677 Per 5.0 0.0 0.1 C 60.1 302.5 20469.0 143.1 392.0 20902 alfPer 1.8 0.5 0.5 1M 58.8 315.2 16320.0 127.7 430.0 21120 omiTau 3.6 0.9 0.7 C 50.7 241.9 36123.0 190.1 3694.0 21278 5.0-0.1 0.0 C 59.6 314.2 16296.0 127.7 464.0 21291 4.2 0.4 0.4 1P 54.4 331.8 14214.0 119.2 373.0 21364 xiTau 3.8-0.1 0.0 2U 51.6 242.2 0.0 0.0 0.0 21389 4.5 0.6 0.5 1U 55.1 330.4 13585.0 116.6 386.0 21428 Per 4.7-0.1 0.0 D 59.7 315.1 0.0 0.0 0.0 [download] -- Mike `-- just,my${.02}`	[reply] [d/l]
Re: data extraction from badly formated txt files by VSarkiss (Monsignor) on Oct 24, 2002 at 19:36 UTC
Those look like fixed-width columns to me. It's much easier (well, relatively ;-) to use `pack` and `unpack` to handle fixed widths because they allow you to break a line down by number of characters. (Check out the pack and unpack docs.)	[reply] [d/l] [select]
Re: data extraction from badly formated txt files by robartes (Priest) on Oct 24, 2002 at 20:22 UTC
This is probably (well, certainly - probably) kicking in an open door, but a good book on stuff like this is our very own davorg's Data Munging with Perl. Among other things, it discusses the care and feeding of pack and unpack. CU Robartes-	[reply]
Re: data extraction from badly formated txt files by heezy (Monk) on Oct 24, 2002 at 23:52 UTC
If you want to avoid using pack/unpack you can often load text files into your favourite spreadsheet, that will allow you to separate the data using fixed width columns. You can then save the data from whatever program you are using in a more suitable "split" format such as tab-delimited. Then you can use.. `@words = split \t, $line;` [download] ..to split on tabs. Thats just an easier way of looking at it for stupid people like me. M	[reply] [d/l]
Re: data extraction from badly formated txt files by Anonymous Monk on Oct 25, 2002 at 01:54 UTC
Thanx for all the help guys. Program works smoothly now.	[reply]