++ nicely stated. The only thing I want to comment on:

But surely: writing regular code does all these things too. So how is the mental process of writing code that belongs buried in deep dark corners different than writing ordinary code? What does obfu do for you? What has it taught you, and how has it made you a better hacker?

The problem with writing regular code is twofold: First, you'll spend a *lot* more time with a small toolbox of idioms for your coding. Second, as your toolbox helps define your abilities, it also affects your vision. You can't easily see how restricted you are for making solutions.

So, while you go about writing regular code, you might easily come up with solutions to all the tasks you encounter. But you're missing the opportunity to solve them better. Every second you spend writing code that you don't need to write is wasted. That time could be better spent thinking about the problem, improving your design, or working on the next project.

A horribly contrived example: Suppose in some bizarro universe, multiplication isn't widely known. It's not unheard of, but it's only used by some elite few. So, if you need to know the area of a square of a given height and width, you might write some code like:

my $area = 0;
for my $h (1..$height) {
   $area = $area + $width;
}
[download]

my $area = $height * $width;
[download]

But after working on the codebase with the "elitists", you start to find it a natural idiom, and forget exactly what the fuss was about. Eventually, unknowingly, you become an elitist--but you now call it "being an experienced programmer". Later, when a "noob" comes by, he asks "What the heck is that? Why should I bother learning that? You guys should stop using such wierd code constructs and use stuff that we noobs understand? Why are you overcomplicating things?" (1)

You sigh, and try to explain. Some will get it immediately, some will take a while, some will have to have it beaten into their heads with a cluebat, and some will move into management roles.

Back to the subject at hand: Writing (or studying) obfuscations help you see more dark corners of the language more quickly, and helps you think in new ways. This helps you as a programmer: When you decompose your big problem into small bite-sized pieces, you'll be able to select the "correct" pieces to break the problem into. (I.e., the largest "trivial" chunks you can.) Ultimately, you'll make a better, smaller, more maintainable (but not by newbies) program.

I don't write obfus myself, as I just don't have that much free time (between my son, my job and my myriad other hobbies). But I do take 'em apart sometimes because they're so educational.

...roboticus

Notes: (1) Sound familiar? ;^)

Kudos to you and the original poster; well-stated and thoughtful. Just to add a touch of gilding to the lily...

I've read (can't recall where) discussions of the difference between what we might call exploratory programming and task-driven programming. In one case the goal is to learn something (feature of the language, a new technique, how a published algorithm works, a design pattern, etc.); in the other the goal is to produce a concrete result (typically under time pressure). While the two goals can mix, they usually are pursued separately under different mental states, which don't mix. A person engaged in one will typically be frustrated by interaction with a person engaged in the other (e.g., thinking of the other as "goofing off" or "being a stick-in-the-mud").

I believe that it's important to have a well-balanced mixture of the two. Although my job involves the creation of code, I feel a day has been lost if I don't learn something (and today I'm usually glad for what I learned yesterday).

Your example of multiplication touches on another interesting contrast: concepts of the problem domain vs. concepts of the programming language. Theoretical mathematics and applied mathematics have developed in a complex dance in which each has taken by turns inspiration and results from the other. Similarly, if the idiom you presented were commonly-enough used, one would expect the DRY principle to drive it into a function. If the function were commonly-enough used, the equivalent operator would become more acceptable. If either became common enough, the concept might "leak" back into the subject area.

All of which is just a round-about way of saying that having fun can be a powerful way to learn (if one is not a stick-in-the-mud).

The darker side of all this is the general "dumbing down" of culture, at least in the US. The media present 15-second sound-bite oversimplifications of complex events/stories. Some employers want to hire inexpensive labor and use them as plug-replaceable factory-line units, even for information-intensive jobs. People of learning and adequate vocabulary are mocked as "eggheads" (or less-flattering terms) instead of respected.

My favorite answer to the above is Dijkstra's response to someone who challenged formal programming techniques as being too difficult for use in the education of "the average programmer". He replied, "What about the education of the average brain-surgeon?"

In medicine, we don't restrict physicians to the narrow range of techniques available to the average nurse's assistant (despite the apparent desires of the insurance industry to second-guess every medical decision and to control treatment by tying payment to their approval of treatment/medication). Similarly, we shouldn't restrict all programmers to the techniques of noobs.

How to get that distinction recognized is another problem.

Excellent! ++!

When I wrote the chunk you quoted at the top of your post, I knew there was an answer like this one but wasn't sure of what it was. Indeed, I was rather sad, because I figured that nobody would figure it out and I would never know. The idea that writing ordinary code tunnels your vision is something that will stick with me. It's something that I've noticed before in other guises - after all, it happens to me all the time when working on actual projects - and never generalized.

In particular, I see it in retrospect when attempting to extend previous work. Having written some code X that accomplishes a task Y, I look back on it and think "Hmm, this should be extended to perform Y+", where Y+ is some related but additional task. When writing X I was following a misguided version of the YAGNI principle and had left that question for later. Interestingly, this holds even if X is the most logical way to accomplish Y alone. If I had considered this all from the beginning, I may have instead written Z, which is the most logical way to accomplish Y only when considering that we may also have to do Y+ (or other related tasks) as well.

• Obfuscation (and golfing, my personal passion) are dual-purpose programs. You are not only making sure that your program correctly solves the problem, but you're also attempting to have the code satisfy a constraint. This helps us write maintainable code because we become used to thinking about providing two deliverables in one package - a program and a work of code.
• The key to mastering any toolkit is to understand the limits of that toolkit. Given that a programming language is nothing more than a toolkit, obfuscation and golfing provide an excellent way of exploring the oft-hidden areas. This gives us confidence that there are tools we haven't learned yet. While I don't use the inner workings of various built-ins that I've learned from golf, I have learned to look at another unexplored area - CPAN. There have been a rash of questions over the past few weeks regarding list manipulations. Many monks have provided answers using code they wrote. All of my answers involved List::MoreUtils.
• As with any other skill, the more uses we put it to, the more varied practice we get. As practice is the key to mastery, the more practice the better. I would be hesitant to hire any developer who doesn't choose to exercise their programming skills in non-work endeavors. They just won't have the practice in.

There is a difference between creative programming and problem oriented programming. Both are creative activities in one sense, but the means and the goal are entirely different.

In "problem oriented programming", you have a particular, application specific goal you want to reach. You have to think in terms of the application domain and the application (or at least it's highly beneficial to do it that way), and somehow find a program that solves the problem you have. The end is more important than the means.

In "creative programming", the goal is fixed. Whether it is fixed to "a JAPH" or "a better plotting algorithm" doesn't matter, because the important thing is the means. You try to find not only a program, but the program, the best there is (with respect to criteria you restrict yourself with). The end doesn't matter as long as the journey is enlightening.

Writing JAPHs lets you focus on technique and methodology. The produce is irrelevant; no-one uses other people's JAPHs, except if they embark on the same journey. The real value is what you go through when writing it.

Mind you, if I have a specific question, there's better places to look for answers.

I try to learn from everyone's obfus by just straining my n00bish mind to attempt to figure parts of them out. I find it's usually a bit more challenging than whatever the solution to my problem-du-jour might be...