Okay, I must be missing something big. You appear to be using $dpf alternately as a file handle (to read()) and as a string (something that you can match regular expressions against). You can't match a filehandle against a regular expression (unless I've missed a major added feature). If you don't pass an argument to pos(), then it works on $_, which you never use anywhere else in your program. I don't see how this can work at all!

I suggest you read the entire contents of the file into a single string and do your processing on the string. Then you won't need \G nor .*.

I want to avoid reading in the entire file, since I am under the impression that I can save memory that way, and it is very easy to let Perl take care of where in the string I am (tell), let me step through it (read) and jump to where I need to be (seek). All of this is possible, I'm sure, in a string, but I'm guessing not as easy.

Whether or not you are allowed to match via a pointer, I don't know. It isn't breaking anything, it just isn't working. I am not sure why it is not working, but I am under the impression that it is matching on the first record ID over and over again, and never moving on.

I could be wrong. I'll play some more and see.

-Travis


---

v2:

Okay, no, I can't match via the file handle. I can do <FIN>=~m//g;, but I imagine that is because it reads <FIN> into $_ and then does the m//g, which means I am reading in the entire file anyway! Nothing saved there, might as well read it all into $str.

-Travis

Yes, you can use tell, seek, and read easily on a file handle. But you can't use a regular expression on a file handle so you need a major change to your approach.

How big are the files? Perl uses lots of memory for most things, so reading even a moderately large file into memory probably isn't go to make much of a difference on memory usage.

In any case, you're going to have to read in a big chunk of the file into a string (say $str) so that you can match a regular expression against that chunk. If you use ($str =~ m/.../g), then you can use $pos= pos($str) to see where you currently are in the string and pos($str)= $newpos to change where in the string to start matching against.

        - tye (but my friends call me "Tye")

m/\G.*[MEH|MED|MMD|MMS|CR1|FR1]/ig
[download]

that should be:

m/\G.*(MEH|MED|MMD|MMS|CR1|FR1)/ig
[download]

or

m/\G.*(?:MEH|MED|MMD|MMS|CR1|FR1)/ig
[download]

if you don't want to capture it. just a minor point, but that could break the code.

also, instead of iterating repeatedly with the loop to find pos(), you could set pos() to $currpos and work from there (you'd have to match two more times, i think). that'll make it a little faster.

jeff

I can set pos() to $currpos? Rock on! I knew there had to be a way to start from where I left off, and not the beginning of the file.

-Travis

> Some of these DPFs occur because a record gets truncated, or goes beyond its length.

So, basically the length field is worthless, but always there? Why not something like this then:

$TLI = "MEH|MED|MMD|MMS|CR1|FR1";
while(m/($TLI)(....)([^$TLI]+)/g) {
  $error=$1; $length=$2; $text=$3; ## Do what you need with them here.
+..
}
[download]

Ideally, the length field tells the parsing program how long the record is. The file is usually broken into its seperate records and put into MQ/Series to be sent to the database.

There are a few things that will cause the file to be rejected. One is, it reads a record (or what it thinks is a record) and then tries to read the next three characters, expecting the next record ID. If they are there, the file is put aside for us to fix.

Now... given what you are saying... will this work like I expect it to?



$RID = "MEH|MED|MMD|MMS|CR1|FR1";

while(<$dpf>) {
    while(m/($RID)(....)(.*)($RID)/g) {
        $RecordID = $1;  $RecordLen = $2; $RecordData = $3;
        if (len($RecordData) != $RecordLen) {
            #ERROR:  Record is wrong length
        }
    }
}
[download]


What I am expecting is: it will step through the file (an example given below) over and over again, pulling each record into $RecordData. Do I need to use \G to get to continue where it left off? Do I need to do anything special for the last record in the file? Why did you use [^$TLI]+?

-Travis
PS: Example file:

MEH0016BUNCHODATA123456MED0019BUNCHMOREDATA456789MED0018MOREDATAAGAIN4
+4568
[download]




---

v0.2: changed while($dpf) to while(<$dpf>). Which brings us back to reading the file into $_. Thank, Tye.

while($dpf) doesn't really do anything. Perhaps you meant while(<$dpf>), but that will only work if you have newlines at appropriate places in the file (which doesn't sound like it is the case) or if you have set $/ to your record separator (but you don't have a record separator, do you?).

        - tye (but my friends call me "Tye")

that last parentheses, ($RID) should be an assertion, (?=$RID). if you don't do this, you will read in that ID and skip it the next time you read something. you don't need the \G unless there is pieces in the data that aren't going to match (it seems like everything in the data is a valid piece of data).

perhaps you want to use split instead of a regex? if every part of your data is getting tested, you probably don't need, or want, a regular expression. split the data on your $rid codes, and test the rest of the data. or since you're reading from a file... read till you hit a $rid code. test what you've read. continue till EOF.

$RID = "MEH|MED|MMD|MMS|CR1|FR1";

open (FIN, "<file.dpf");

my $filecontents = <FIN>;
$filecontents =~ s/($RID)/\n$1/ig;
my @filecontents = split('\n', $filecontents);

my $count = 0;
foreach my $record (@filecontents) {
    print ++$count . "\t$record\n";
}
[download]

    local( $/ )= undef;
    @records= split /(?=($RID))/, <FIN>;
[download]

Note that you need to set $/ if you want to read in the entire file at once.

So a record can never have, for example, "MEH" in the middle of it? What kind of data is in these records that you can guarantee that these record IDs never appear in the middle of a record?

        - tye (but my friends call me "Tye")