There is a good reaason why search engines don't allow full regex searches--they are simply too slow. You'd be better only allowing keyword searches.

But, if you decide to try the big file route, a couple of things that will help are:

• When concatenating your files, remove all the newlines so that each file becomes a single line prefixed by the path information.

  This makes searching for phrases that span lines much simpler and much, much faster.

  If you need to obtain position information, re-search the actual files once you've located them from the master file.

• If you have multiple cores available, split the master into as many parts as you have cores and search them in parallel.

  If you can distribute these parts across different spindles, you'll gain the maximum advantage of the parallelism. Otherwise, you may not see much benefit due to head thrash.

  None the less, by making each file a single long line in the master, the ratio of cpu to IO will be greatly improved.

• If your OS/filesystem allows you to override the default IO buffer size, increasing it to encompass the average (or maximum) line length may yield benefits.
• Make your master file(s) contiguous if your filesystem allows that.

If you can limit a first pass to keywords only, then building an inverted index would be the fastest way of trimming the dataset. You could then allow a full regex search to be run on the subset of documents.