I'm still in the progress of making our company's app fully scalable. The only outstanding issue is the file server: I have yet to find a good solution to mirror our nfs server. Our data is in de low terabytes at the moment, and growing by a couple of gigabytes per day. A typical rsync run takes a couple of hours, so that's not a real-time solution. Does anyone have good pointers on this?

As for the rest, everything is redundant. We have 4 tiers:

1. Static front-end Apaches
2. mod_perl Apaches (*)
3. Replicated MySQL servers
4. A big fileserver

(*) Actually, this layer also has a number of Apaches setup to do straight cgi, for those long-running requests like file upload.

Here's some links I found useful in setting this up:

• The front-ends are clustered using RRDNS
• The tiers 1. and 2. are connected using Randomized Rewrite Maps and the Reverse Proxy setup
• The MySQL servers are replicated as described in their Howto

So that still leaves me with the file server. I haven't been able to investigate clustered filesystems such as GFS, Coda and others yet, but there may be a solution there.

And the other big thing I'm not trying to worry about just yet is how to mirror across data centers. I'll cross that bridge when we get there :)