It would be far better if Perl had internal support of some kind for this, but look at the following:

(?=\z(?{ die "end of string" })|)
[download]

This convoluted construct will cause the match to die if you reach the end of string at a given point in the RE. You can trap this in an eval. Take an RE and sprinkle these heavily and you can guarantee that if the RE reached the end of your current text at any point during the RE match, then you will find out about it and know to grab more text. You will need to tokenize the RE carefully to figure out where to insert the end of string tests. (And don't forget to remove them if you know that there is no more input to add!)

I looked around for a simpler way to do this. I didn't find one. It would be ideal if there was a regexp modifier to set "end of string" behaviour. But nobody seems to have implemented that...