My understanding is that your application should not contain code exists only there to make the tests pass. If you do all the work in your test, then it doesn't matter if you're working offline, your tests should run without complaining that they can't connect to the server. Keeping an LDIF file in your tests makes a lot of sense in the separation of code. To counter problems with slow tests, I suppose you want the setup/teardown to be efficient, the LDIF only as big as necessary and to consider Continuous Integration to run the full test suite automatically.

Is your LDAP sitting in your application or is it external? Would you be able to post some test code snippits? I'd like to see how you do it.