There are challenges at different levels. You mention "building infrastructure". Some of the stuff I have enjoyed doing most is of that sort, although more often tool building than library building. It is especially satisfying when everyone on the team uses the result every day and it works so well that it has pretty much disappeared into the woodwork.

Finding elegant solutions to problems of any size. For example, I enjoy challenges related to golf. Not golfing code to obscurity, but golfing it to a point of elegance where it is clean, succinct and clear. Many answers to PerlMonks questions include fragments of that nature. Not always the main part of the answer, but often a clean way of setting up the preconditions for the core part of the answer.