I'm not too sure I have a very clear view of what the skillset of the "average programmer" looks like. I'm fairly sure if you took the average skill level of everyone who work with Perl, it would be pretty low because of all the poor CGI programming out there. (I'm also sure it'd be worse without the Monastery, where everyone tries to improve their skills.)

I imagine the spirit of your post could be phrased in something like "What little corner of Perl have you worked hard to master that other Perl programmers might have overlooked?"

So, in the spirit of that question, my work in Corpus Linguistics has forced me to be a fair hand at dealing with data munging. We constantly need to get data from some raw form into something that we can run through a natural language tokenizer, then through a tagger, then do some post-tagging cleanup (parsing), and then export our data to a MySQL database.

I recently rewrote our tokenizer using Parse::RecDescent. The net result is that the tokenization is slower, but it the accuracy and recall are much higher. A lot of that improvement had to do with re-thinking/re-factoring the algorithms I was using to do this.

Another thing that is really necessary and is often overlooked (or neglected) by Perl programmers--especially in the US--is dealing with various encodings that are not ISO-8859-1. It saves us work in the long run if our scripts handle ISO-5559-15 and UTF-8 just as well. So perllocale is one thing I've really had to master.

The (Perl) programmers I know who do lots of parsing tend to avoid Parse::RecDescent because it's so slow. They all use Parse::YAPP which is must faster. You might want to give that a spin instead. It's meant to be yacc-compatible, so if you're familiar with that already then you're in good company.

It would be interesting to see a real-world comparison between the two. (hint hint :-)

Yes, indeed. :) I think both bart and Corion suggested I use Parse::YAPP, so you're in good company. The tokenizer was my project for last week and I've got another project for the next fortnight, but I will definitely give it a burl when I get the chance. After all, it works (and we can leave it running overnight). ;)

I'm not really from the yacc/bison/(f)lex crowd, but I am familiar with Benchmark, so maybe I'll take a hint and grant your wish.

--
Allolex

The term "average" is difficult to apply to Perl. The skillsets among those programming Perl are too varied for the average to be meaningful. I rarely write any perl -e's (and when I do, I tend write them in a strict-safe manner, even though I'd get the job done faster not doing so), but I'm often creating CGIs and OO modules. OTOH, someone with a background in system administration may write nothing but perl -e's.

Update: s/to/too/ in second sentance.

I do think the use of perl -es is a big domain of sys admins and less for CGI programmers. That doesn't mean they can be any less useful for CGI / OO programmers. (Note, I am using the term perl -es liberally and to refer to any quick and dirty script that makes work lighter and can be thrown away -- but saves quite a bit of time).

To give you a good example, one of my clients got bought out, and wanted me to change widgetscorp.com to widgetscorpinternational.com, and "Widgets Corp., Inc." to "Widgets Corp. International, Inc., A Subdivision of International, Inc." within all of their web pages and CGI scripts on their site. (Several hundred pages combined, not all of which I had done. They however wanted me to change it all under the technical support I was supposed to give them for the site.).

Now I really wanted to keep them as a client, even though I would have had every right to tell them that something of this magnitude was outside the scope of the contract. (I mean, there wasn't anything wrong with the CGI scripts I had written, i.e. bugs, and it wasn't my fault they were bought out!) But, the idea of going through all of their web pages and scripts -- even doing a search and replace -- to change them didn't thrill me either, even if I were to be paid for it because it was grunt work.

Enter Perl and File::Find to the rescue. 5 minutes and I had a script to traverse an arbitrary directory, run an s/widgetscorp\.com/widgetscorpinternational\.com/sgi, and I looked like a hero. Even better, I could e-mail them a copy of the script, and then they could use it internally on their files.

Needless to say they were tickled pink for 5 minutes of my time -- which was well spent because it helped cement my relationship with Widgets Corp International when they were being shaken up.

Looking like a hero++

Pretty much any Perl programmer can benifit from perl -e. The question is if you need more than that to do your job. CGI and application programmers certainly do, but sys admins might not.

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

: () { :|:& };:

Note: All code is untested, unless otherwise stated

Don't tell them the secret! Now they won't need you. ;)

I use perl a lot for global search and replace, for example, perl -i -pe 's/before/after/g' *.txt . I use it so much that I have pp aliased to 'perl -i -pe'.

Regarding the original post, one "easy thing" that I think it's very much worth knowing and that some novices overlook is the appropriate use of grep and map. Sometimes you can save lots of time and typing and write clearer code with grep and map.

quite true, quite true.

i asked a colleague for an opinion about which module to use for a task ( in this case LWP vs. WWW::Mechanize ) and his reply was "I've never used either". and he's supposed to be a consultant 'helping us catch up'. ( needless to say that concerned me a bit re: his 'consultant' qualifications ).

In all, it's all easy at this point. I know I can get there knowing the available tools.

Things I have done in the past that are a little rusty, yet I could pick back up in 5 minutes -- Databases, socket programming

Things I've beat to death recently -- Data serialization between various apps, Perl/Tk, build systems (a close cousin to system administration, but not quite the same)

Things I haven't completely explored yet, but easy just the same -- large-scale O.O. in Perl (haven't needed it for what I do at work other than some module exploitation). This is allright though, since I'm biased against most large-scale OO designs because they make some things too hard/complex, and I use Perl to escape pain, not create it.

I've only found a few things "hard" per se, like trying to write my own Parse::RecDescent grammar. The hype in this monastary towards this module makes me think a few people know what it is, but most are using it for very simple record parsing.

Summary: Perl makes the hard things easy and the impossible things possible. That's why I like the language. A beginner is going to have trouble only in learning the ins-and-outs of the language, how things are done in a Perlish way, and how to check his desire to go into full-hog obfuscation-show-off-mode. Once he gets to the level where he understands the dark corners, it will all be easy, or at least doable.

Now ask me to do half the Perl things in Java (a language that I know well, btw) and I'd have to say many of those things are still quite hard and require building a lot of infrastructure. The "Swiss Army Chainsaw" is infinitely useful everywhere!

Other responses have latched on to perl -e and that's probably one good avenue to explore. Here's a real-world example for you. I recently downloaded a bunch of files from a Windows system and uploaded them to a FreeBSD system, only to find that the spaces in the file names had been encoded so they were now '%20'. I had to rename three directories, each containing six files.

After wimping out on the first two directories (this was time critical stuff and it was after midnight, a great time for making dumb mistakes, especially after a jam-packed 11 hour work day) I wrote

perl -e 'while(<*.mp3>){$orig=$_;s/%20/ /;print"$orig -> $_\n";}'
[download]

perl -e 'while(<*.mp3>){$orig=$_;s/%20/\\ /;print`mv -v $orig $_`;}'
[download]

So what did that require that I wouldn't have had a clue how to do as a beginner?

• Remembering to use ' to quote a one liner so that I can write print statements with " so I can use variable interpolation;
• Making use of the glob operator to get all of the file names using a particular mask;
• (A Unix/Linux one: the -v switch makes a command 'verbose', useful when you want feedback about what it did);
• Remembering to save the original name when I want to have both the original and the new name;
• (A programming thing: Carefully review your code, thinking about exactly what it's going to do when it runs)

The other thing I can recommend is, get familiar with CPAN. Learnt to use the CPAN shell to install modules. Someone else has spent a fair bit of time and intellectual energy creating this varied collection of masterpieces, and there really is something for everyone.

I tried to use PDF::API2 and could not get it to work. I gave up on it for a while, when discovered that good old pair Networks had already installed PDF::API2::Lite and got it to work. The bad news is that the documentation sucks; the good news is that I can more or less get it to work, since creating PDFs is one of the things I need to do to support my singing hobby.

I still want to get the module working on my home system, since pair's CPU limiting doodads cut in whenever I run the Perl debugger on that module, but that's miles better than not being to produce PDFs at all.

I guess to a point it's all subjective as to what you find easy or otherwise.

In my experience two things that tend to trip up "new" perl programmers are regexes and Object Orientation.

In the most part neither of these is due to difficultly - more to the reputation they have as "hard to understand". I've know seasoned programmers to stay well clear of OO writing OO modules simple because of the perception.

My skills took a step up (from very poor to modestly mediocre) when I worked through some of the network examples in Lincoln Steins' "Network Programming with Perl". First it showed the concept of "more than one way to do it" and second it helped contrast OO and Non-OO methods. It also created a very useful set of tools for inter-system operations which are powerful.

I would use the search command in CPAN to look up topics of interest to your group. The math routines are interesting - try the MATH::FFT as a well defined example of something very useful.

This site has a bazillion code examples. Grab some, run them and modify to your tastes. Note also the use of Secret Search and Super Search.

Here are some additional and useful examples.

Try This

Diskcrash