A BerkeleyDB tied hash temp file might be a good choice. Your hash key would be your sort criteria, the value would ideally be the regexp replaced fields, unless you really need the original data for something else.

Once you're done reading/storing the SQL results, it's a simple matter to read the hash back in sorted order, doing the printf formatting and less piping then.

You might also consider saving aside the BerkeleyDB temp files as an expiring cache if you get a lot of common queries that don't necessarily need absolutely up-to-date data, and/or user can specify when they do need the latest data. You're doing a lot of I/O here, it's not gonna be fast. Some users might be willing to trade data currency for response time.

Is there any specific reason you're not making the SQL query directly from perl? Assuming its possible, that would likely reduce the complexity a great deal, and increase the reliability. Avoiding data problems introduced by parsing raw text output from another program is always good.

Thanks for your response Dave. The reason I am not doing SQL queries directly from Perl is that I do not have permission to access the DB directly. The person who runs the DB has created an intermediary tool that allows me to pass in SQL queries and it then sends back all the results like this:

Input:

sql_query_tool "SELECT * FROM table..."

Output:

Field1||Field2||Field3||Field4
Field1||Field2||Field3||Field4
...
[download]

Also, I have sent my sort criteria to the DB using the query tool, and it works fine. The problem is that certain fields are blank in some cases, so it causes the sort to get screwed up in the following way:

   category     type1    type2     type3

1. fruit        apple    orange    banana
2. fruit        apple              tangerine
3. deserts      pie      cake      icecream
[download]

When I try to sort the fields in order of category -> type1 -> type2 -> type3 I have a problem. Sometimes the type fields can be blank. What I want is for these blank fields to be ignored by sort and the next field to be sorted on. Instead what currently happens is the blank field is sorted on which always makes it first. In the example above record 2 is missing the type2 field. When I do the sort it will put record 2 before record 1. Then when I print out the final data it will look like this:

fruit    apple    tangerine
fruit    apple    orange       banana
[download]

So when printed out tangerine comes before orange, because of that blank field being sorted on. This is why I came to the conclusion that I could not sort using the DB, and needed to parse the data first myself and then sort.