Calling it a "directory" is a little misleading, because it's a HTML page, not a directory (in, for example, the opendir and readdir senses).

You'll need to parse the HTML and look for links to .txt files. I suggest HTML::Parser; you can try to do it with a regular expression, but it can get be incredibly hairy and lots of unneeded work (and I wouldn't suggest it).

Then you can use LWP to get each of the links (to text files) that you find and process them as you will.

hdp.