Many tasks are just plain complicated on their face. Writing an SQL generator that takes into account arbitrary schemas, arbitrary constraints, and selective denormalization, then builds the correct optimized SQL is hard. It is correctly broken out into functional areas. One of those areas is the use of a directed acyclic graph (DAG) to represent the schema. I certainly didn't write that code (though I ended up rewriting it to suit my needs). But, that was a conceptual black-box interface. Although I know nothing about inverse indices, I'm pretty sure that, like all other CS problems, it decomposes quite nicely into areas that are generalizable. Anything that's generalizable is a conceptual interface that is another distribution.

Yes, your data is going to pass through different distros, and that's ok. The big thing to focus on is the idea of separation of concerns. My SQL generator didn't need to know how a DAG worked, just that if you hit the red button while pulling the green lever, the blue light will come on. I suspect that there's a lot of stuff on CPAN you can certainly reuse, reducing your coding (and testing) burden.

IMO, developers who work this way are not terrorists who hate our freedom and are out to destroy our way of life. Good software which does useful stuff can be written in many ways.

Absolutely (*grins*) true on both points. However, read my signature. Good software can come from many places, but it has two very basic criteria - it works and someone else can safely modify it. If it doesn't meet those two things, it isn't good software. And, frankly, that is an absolute.

Now, how do we meet these criteria? Well, the most efficient way (thus far) is TDD. How do you do TDD with a complex system? By mocking up your interfaces. You test your intermediate items by mocking up their dependencies. Then, you have some system-level tests which exercise the system as a whole without mocks, and you're 80-90% of the way there.

How do you avoid tests for the intermediate processes which rely on the innards of what ought to be a series of black boxes? The answer is... there's no easy answer. Mock objects help. But there's going to be a fair amount of waste...

Waste? I don't see waste. Yes, you will have to keep your mock objects current with the spec of your intermediate sections. However, while specs may grow, and quickly at times, they should change items very slowly. Otherwise, it's an incompatible interface change which should happen either with a new major version or in alpha software. Anything else is churn which screws you up no matter what paradigm(s) you're using.

. . . the recommendations are inappropriate for my current task, which is porting an existing library not designed from the ground up according to the principles you set out.

You're rewriting the library from scratch, period. You're using a different language and you have access to different libraries and language features. You may be preserving the same API and functionality, but it's still a complete rewrite. Porting the docs is all well and good, but you still need tests written against the spec that your code will first fail against, then pass as you write the minimum necessary. I have no experience with Lucene, but I am 100% positive that there is cruft in that codebase. By rewriting against the spec using the existing codebase as a reference, you most certainly can use TDD. In addition, you can probably end up with several new distros to add to CPAN that aren't directly usable solely for reverse indexing.

I'm doing a very similar project in Javascript, porting the Prototype library to JSAN. Instead of just throwing it up there, I'm converting the innards to JSAN distributions, cleaning them up and renaming them. I'm also keeping a compatibility layer so that existing users of Prototype (such as Ruby-on-Rails and Catalyst) can convert over to JSAN with little change in their existing codebase while taking advantage of the better code. I suspect you will find that you can do the same.