Just my 2 cents :-)

I have a policy myself which is, whenever a shell script goes beyond 3 lines I stop and rewrite it in Perl. But one thing I will be very glad to use will be the Perl 6 ==>, since it's somewhat boring to chain piped commands from Perl 5...

If process startup time is significant. Note that shell has a lower startup cost than Perl.

I'd not include that item. It sounds very much like premature micro-optimization. Recently someone was ranting about 'sed' being better in some cases than Perl because it starts up faster. My testing (on two very different systems) showed Perl starting up about twice as fast as 'sed' and both starting fast enough that the difference was unlikely to be noticed. I think you'd be hard pressed to properly document how little your shell script can do and still keep this "advantage" over a Perl script.

I decided to learn Perl because I was unsatisfied by what I could achieve with batch and shell scripting. Granted, I knew only a little shell scripting, but I did a lot of batch scripting. In particular, flow control and error checking are difficult and verbose in a batch script. Also, there are too many things that you just can't do without firing up a C compiler and writing a utility or searching for a ported unix tool. The unix shells are better off, since one rarely needs to hack up a custom tool, but one needs to learn awk and sed and find and a thousand other little programs to do really cool stuff. Perl's portability is key.

The other thing that sold me on Perl over batch/shell scripting, is Perl's flexibility--once I built expertise in Perl, I knew I could apply it for many things other than the original problems I was trying to solve. Since you've already got a lot of Perl in your environment, using it in place of batch and shell will "deepen your bench"--more people will have and use skills that apply across your environment.

As to reasons to write something in a shell or batch file, I wouldn't go for a checklist. I'd say simply that "Everything should be written in Perl unless there is a compelling reason to do otherwise." Too often I've "known" that something need not be portable or will always be small, only to be proven wrong. Neither "it doesn't need to be portable", "it's really short" are reasons to write a batch or shell script instead of Perl. If either condition is false, there needs to be a very strong argument to go with a batch/shell solution over Perl.

I agree with tye that the third bullet (process startup time) is better off not being mentioned. It's hard to imagine a case where startup time would be both so significant and so reliably favorable toward shell scripting rather than perl. On the other hand, it's easy to imagine cases where a shell script's tendency to invoke lots of sub-processes (e.g. running sed or grep inside a for loop) places it at a serious disadvantage relative to a perl script that can easily do the same task within a single process. I know you already cite this as a reason for using perl instead of a shell script -- it's just that this third item in the checklist could lead some readers astray.

Also, I would elaborate on the first item in the list: a unix environment in which Perl is not available would include (very significantly) situations where unix is being run in single-user mode, or in similarly "defective" conditions (restoring from backups, etc) where any of /usr/bin, /usr/local/bin, /usr/local/lib (and so on) are not present -- i.e. where there is nothing else but the shell (and the other stuff in /bin) for getting things done. I think this consideration will help people to focus on the appropriate domain and scope for shell scripting.

And for the shell scripts that are to be written for situations like that, it should be strongly emphasized that the second checklist item (keep it simple and short) is likely to be a matter of "life or death."

The main reason is error handling. Exceptions, warnings, controlled failures? Forget all that in shell script, one bad error and it fails, no warning, no log.

A secondary reason is portability. Let's take "echo". Depending whether you're on Solaris or AIX or Linux my guess is you're wondering whether to run /bin/echo or builtin echo or echo -e or whatever to get a \n translated to carriage return. Have you ever looked at the shell script behind GNU configure? That says it all, and what it's saying is "should have been written in Perl!".

Okay, backaways when you might have perl4 or perl5.05 or perl 5.06 out of the box with your distro it made sense to use shell for more portability. Since the advent of perl 5.06 the language has been stable enough (more so than sh/csh+Unix of choice) to be preferable.

Regards, Peter
http://perl.dragonstaff.co.uk

IIRC you can set env vars in a .bat script that will persist after it exits, in the same command window. This is used, for instance, for setting up Visual Studio paths (the venerable vcvars32.bat script). I don't think this can be done in Perl ?!

Your variables will stick around unless you do a setlocal, default behavior is to screw up your environment. I just love DOS.


TGI says moo

One job I had many years ago was writing a standard installer for a Unix app written in C. I used shell. But it had to run on AIX, ICL Unix, DEC Unix, Sys V SCO OpenUnix and BSD. Very little standardisation across shells on those platforms, nightmare to work around the quirks of different machines.

Let me give another example of shell suckiness. Write a script to run FTP, check the output and if it fails do some rollback action. Perl, easy (Net::FTP). Shell, you run, grep and pray. Again, different platforms, different ftp command line syntax... Linux, Solaris, argh.

Regards, Peter
http://perl.dragonstaff.co.uk

--
peterdragon

My experiences are different. I once rewrote from scratch an installer/upgrader for the product the company I worked for produced. The result, a 2.5k shell/awk program, using isql (Sybase command line tool) to do its interaction with the database. Initially developed on Solaris, but I knew it had to run on HP-UX and Windows NT as well. The porting took less than 30 minutes - I was using an option to grep HP-UX didn't know about, and Windows NT (which had a Unix toolkit installed) didn't know about /tmp.

Later, I've used shell scripts initially developped on Linux without much problems under Windows/cygwin.

Of course, you may say, Windows doesn't come with a Unix toolkit. You are right of course. But Windows doesn't come with Perl either, so whether you want to use Perl or shell to do your cross-platform scripting, you will have to install something.

Why I didn't write the installer I mentioned in the first paragraph in Perl, you may ask. Well, at that time, Solaris didn't come with perl, so we couldn't assume perl was available. And I hadn't convinced the company I worked for that we should just bundle perl with our product. Yet. A year later, I had to write a different installer. By then, we did bundle perl with our product, and that one was written in Perl. By the time, we had dropped Windows as a platform so I don't know whether it would have worked on Windows.

I generally concur with that opinion, although I would have stated it in far fewer parargraphs and bullet-points and I would counsel you to do the same. (“Preachers” tend to get crucified, even if their message is immortal.)

Simply stated, I think that “shell scripts” are intended to be just that:   a moderately sophisticated way to tie shell-commands together. Nothing less, and nothing more. Shell-designers weren't trying to invent a general-purpose programming language (like Perl) since such languages already existed. Instead, they gave you the ability to use any command-processor you like, through the simple mechanism of (#!commandname) “shebang.”

A shop should agree upon a working-standard and then stick with it. But they should make sure that they're using the right tool for a particular job. All you'll get for the wasted-time that you just spent proving that a wrench can be used as a jackhammer is maybe “w00t! w00t!” (While you're perfecting your curious monstrosity, your colleagues are munching fish-n-chips down at the pub.)

But they should make sure that they're using the right tool for a particular job.

Using the right tool for the job is important; I've worked at shops where people wrote thousands of lines of C to accomplish what could be done in ten lines of Perl. Yet companies need to set a practical limit on the number of supported languages when they commit to maintaining code in these languages over a period of many years. After all, mastering, as opposed to dabbling in, a language, and its libraries, and its community, takes a lot of time and effort.

What is a sound practical limit on the number of languages a company can comfortably support? I don't know, and it depends on the company, but my perhaps conservative opinion is that my company should support just one "fast" statically typed language and just one "dynamic" language. Maybe two. Any more than two would be a mistake IMHO. For example, I feel writing part of our system software in D, another part in Haskell, another in Erlang, and another in C++ would be a strategic mistake, even if each was indeed the "right tool for the job". Ditto for writing in a combination of Perl, Ruby, Python, and Lua.

Update: Even a company as big as Google only allow three languages to be used for production code, namely C++, Java and Python.

Portable anything is difficult, the question is how difficult.

If you give me the option of porting a shell script and a Perl script to a win32 environment, I'll take Perl. It's still not a fun job, but at least it isn't a painful job.

eyepopslikeamosquito writes:

"Shell scripts tend to be insecure. Running an external command in a global environment is inherently less secure than calling an internal function. Which is why most Unices do not allow a shell script to be setuid."

Others are interested in keeping it alive. Indeed, Perl ships without setuid compiled on some distros.

A lot of the negatives built into some of the shells can be seen in differing degrees in perl. It's interpreted nature means it can be slow. It's eclectic feature set means it can be difficult to audit. It's cooperative nature means scripts can call questionable binaries. It's comprehensive nature means it's footprint is large. So it would be dangerous to attack shells based on their architecture.

"Shell scripts, being interpreted and often having to create new processes to run external commands to do work, tend to be slow."

Perl is also interpreted. Although I think that is not the crux of your argument I would chose different wording.

Perl is not interpreted. At least it is not in the same sense as shell. Shell scripts are source-level interpreted. Perl programs go through a full-scan compilation into another form which is then executed. It isn't translated directly into machine code, but syntax errors are caught at compilation time and not run time. The process does happen generally every time you run a program, but that does not mean it doesn't happen.

Nearly every useful language has the ability to execute external commands. Most shells rely on that for most of their utility. If you think Perl is insecure because it can call an external binary, then what of shell, which often must?

Also related to how shell is used is your footprint comment. How is Perl's footprint compared to every Unix command available on every version of Unix? Undisciplined shell programmers can use every CLI-based program on the system to get their work done. They often must use the search path to attempt any kind of portability, and can be effected by other environment differences. An undisciplined Perl programmer who uses every dark corner of Perl can at least expect some compatibility among different installations of the same version. If you're depending on external tools for most of your functionality, you can't count on much of anything.

Shell doesn't have an eclectic feature set? Again, you're talking about the entire installed command set of whatever machine you're on, often depending on the search path and environment variables.

Perl has its weaknesses, but I don't think you're assessing them fairly here. Perl and most shells are worlds apart.

As mr_mischief has already pointed out, Perl scripts are first compiled to an internal form, ensuring syntax errors are caught at compile time rather than run time ... and considerably speeding up script execution at run time.

To give a specific example of where shell can be slow, consider the common task of running an external command recursively on all files under a given directory. In shell, you could use the find command with its -exec option, but that results in a new process being created for every file. I've measured cases traversing thousands of files where such an approach proved to be hundreds of times slower than using Perl's File::Find module in harness with a Perl function performing the work of the external command. The performance difference is especially noticeable when the external command doesn't do much work. Of course, the way to solve this performance problem in shell is to use find in harness with xargs, but there are portability pitfalls for the unwary, namely when the filenames contain spaces or other unusual characters -- as indicated in a response to a 2002 gnat use.perl.org journal entry. Update: broken link, these are the commands:

find . -print0 | xargs -0 ls -l                 (GNU only)
find . -print  | sed 's/ /\\ /g' | xargs ls -l  (works everywhere)
[download]

Found an archived link: gnat use.perl.org journal entry (see response by oneiron)

As eyepopslikeamosquito and mr_mischief point out there are important differences between perl's interpreter and others. However I was attempting, albeit poorly, to point out that simply saying "shells are bad because they're interpreted" could backfire if you are trying to sell perl. My concern fell out of some confusion I had when reading the OP as to whether, briefly, he was talking about shells or perl in regards to interpreters.

As a separate caution, those PHBs unfamiliar with perl can be uneasy with perl's power and diversity. It may be best to soft-sell these points and concentrate on portability and organizational issues where having 'one code to rule' offers clear advantages.

I apologize that my previous post was cryptic and confusing and I thank eyepopslikeamosquito and mr_mischief for setting me straight. I am well aware, however, of perl's capabilities and I love beautiful perl code like the rest of the Monks here. However not all of perl's features are optimized for systems programing and I find some of it's features to be quite clunky in those applications. I think the Perl Evangelist would profit well in gleaning from fellow Monks which characteristics offer a clear, and easy to communicate, distinction with other shells. The likewise said evangelist would profit well in avoiding those comparisons which could be misconstrued by the casual observer as symantic sugar.

IMO, this is a classic case of horses for courses - shell script is marvellous (C shell excepted of course - Tom Christianson. http://www.perl.com/pub/language/versus/csh.html) for *NIX systems programming - for all else, perl is, by far & away, the front runner.

Just my 10 p'worth...

Well, that quite depends on the setup. Many Linux distros use one big partition by default anyway (or at least have /usr on the root mount), so once you have anything, you have perl (assuming it's in /usr/bin/perl). And traditionally, the root mount contained /bin with just a small set of programs, just enough to get to the stage to mount other file systems. Anything interesting would be in /usr/bin anyway. Furthermore, mounting other file systems happens pretty early in the boot process anyway - so most rc scripts will have perl available, even if perl is not on the root mount.

But I've also worked for a company where we used Linux boxes that only had 2 Mb of memory, 25 Mb disks, and for which the OS had to be installable from a single floppy disk. Needless to say, said boxes didn't have Perl, although I used Perl a lot to create the distros.

...there's an awful lot "goes on" after getting to single user run level...i.e. before the rest of the file systems are checked & mounted.
A user level that continues to overstate my experience :-))

Writing portable shell scripts is hard. I don't think it's hard, although it's easy to write something that is not portable. But you have several options available to make porting easier. First is to limit yourself to the POSIX standard. Most Unix vendors support at least POSIX complaince for their shell tools. The second option is to use the GNU tools, and install the GNU tools on all the platforms your program needs to run on. GNU tools have been ported to all major Unix platforms, and most run on Windows as well. Is each external command available on each of our platforms? That's a valid question to ask when porting a shell program, but an equally valid question is to ask whether each Perl module you use is available on each of the platforms you use. This may usually be true for pure Perl modules, but is not always true for XS modules. Specially not if said modules use third party libraries.

Shell scripts tend to be fragile. Running commands and scraping their output is an inherently fragile technique, breaking when the format of the command output changes. My experience is that the behaviour of shell commands tends to change less than the behaviour of Perl modules. Sure, a shell command may output something different if you upgrade it, but a Perl module may change its behaviour as well if you upgrade it. And Perl modules don't have a standard - shell programs do: POSIX. And then there's perl itself. In the 25+ years of programming that I do, I cannot recall a program breaking because of an upgrade of the shell. An upgrade of Perl always tends to break at least one of my programs in some way.

Shell scripts tend to be slow. That's 'the pot calling the kettle black', isn't? Agreed, the shell doesn't shine when it comes to speed, but neither does Perl, does it? Luckily, with modern machines, it's usually the disk or the network (or the memory) that's the bottleneck, so the relative slowness of the language itself doesn't really matter. But in the few cases where it did, I'm always amazed by the speed of C compared to Perl (every now and then I rewrite a Perl program in C for speed reasons).

Shell scripts tend to be insecure. Running an external command in a global environment is inherently less secure than calling an internal function. Eh, not necessarely. Languages don't make programs (in)secure. Programmers do. And frankly, I have more trust in what my vendor puts in /usr/bin than what I download from CPAN. I do not agree that running an "external command" is "inherently" less secure than calling an "internal function". And I've no idea what you mean by "global environment".

As a programming language, shell is primitive. Shell does not have namespaces, modules, objects, inheritance, exception handling, complex data structures and other language facilities and tools (e.g. Perl::Critic, Perl::Tidy, Devel::Cover, Pod::Coverage, ...) needed to support programming in the large. Up to version 5, Perl didn't have namespaces, modules, objects, inheritance nor exception handling. And C still doesn't have them. And depending what you mean by 'complex data structures', C doesn't have them either. As for Perl::Critic, Perl::Tidy, Devel::Cover, Pod::Coverage, I disagree that they are needed - in fact, I wouldn't want to use Perl if they were needed to program in Perl. And in the history of Perl, they are the new kids on the block. People prefered perl over shell long before those tools were first created.

If you're sure the script will "never" need to be ported to Windows. Been, there, done that. Shells (and shell like tools) have been ported to Windows, just as Perl has. And remember the time the perl port on Windows was different from the one on Unix? At that time, that wasn't true for some of the shells. "Oneperl", the perl that merged the Unix and Windows ports only dates from 1998; macperl merged even later than that (and that only because MacOS went Unix).

I do not agree that running an "external command" is "inherently" less secure than calling an "internal function". And I've no idea what you mean by "global environment".

By global environment, I was referring to environment variables (e.g. PATH, IFS, CDPATH, ENV, BASH_ENV, SHELL, TZ, LD_LIBRARY_PATH) and other elements of the execution environment (e.g. umask, inherited file descriptors, temporary files) that are a common source of exploits by malicious attackers. Certainly, executing an external program securely is not trivial: there are many, many security exploits to consider and guard against. That's why I stated that calling an internal function was inherently more secure -- because all these many and varied exploits need not be considered.

To give a specific example, most shell scripts tend to use the (potentially insecure) $HOME and $SHELL environment variables to ascertain a user's home directory and shell, while a Perl script can get this information via the more secure (and more reliable) getpwnam internal function.

That shell scripts tend to be insecure is widely known and acknowledged; see, for example, FAQ: How can I get setuid shell scripts to work? and perlsec, which opens with:

Unlike most command line shells, which are based on multiple substitution passes on each line of the script, Perl uses a more conventional evaluation scheme with fewer hidden snags. Additionally, because the language has more builtin functionality, it can rely less upon external (and possibly untrustworthy) programs to accomplish its purposes.