Whether or not Perl is "scalable" can depend upon how you defined "scalable". Do you mean "large projects can run using Perl as the primary language"? I would argue that they can. If you mean "many programmers can work on the same Perl project", then scalability can depend upon the strength of your organization. Do you have strong programmers and good standards in place? Do you have strong project management? These issues, from what I can see, cause problems to crop up quicker in Perl shops than others. However, it's important to note that these problems occur in any shop regardless of the language.

Perl, however, has a unique approach. Rather than increase the power of a team to work together (which Java can do), it can increase the power of the individual programmer allowing a smaller team to be more productive in Perl than a larger team working with another language. This can lower overall costs and increase communication (because there are fewer communication paths) and this is a benefit to the success, and ultimately the scalability, of almost any project.

I'll finish this by posting an email reply that I made to someone who was concerned about this issue.

I don't know all that much about programming. I am under the impression that Perl is fine for fairly low traffic CGI, but C would be needed when the site scaled beyond a certain amount of traffic. Please correct me if I am wrong.

That depends upon exactly what you are trying to do and what technologies you are trying to apply to it. For example, I know of a company (that I cannot name) where the programming team was working in C++ to develop a large Web site to manage financial data for customers. A friend of mine was their QA director. After I learned a bit more about the company, I discovered that they had quality management, a sharp QA team and solid financial backing. Further, they were offering their employees generous stock options. I urged my friend to quit the company.

This sounds a bit ridiculous, but the problem was due to their choice of programming language (languages like C and Java often suffer the same problem). It takes too long to develop in them. Quite frequently, using a language like Perl allows you to finish your program and move it into production before competitors have even gotten their first compile. With my friend's company, it took them two years to get their prototype working. This means that if it was successful, competitors using a language such as Perl (or Python, Lisp, or other language that's quick to develop in) would likely be able to duplicate their results in only a few months. Further, any new features that my friends company could produce could be duplicated in a much shorter time frame at a fraction of the cost.

My friend is now looking for a job. He was convinced they would succeed and every time I talked to him, I was impressed with what they were trying to do, but I was convinced that they made the wrong technology choices. The company has gone bankrupt because the technologies they chose simply took too long to develop in. (side note, not in the email: there was nothing that this company was doing that required C++. It was vanilla CGI with a powerful database backend that DBI would have handled nicely. And yes, I know that the DBDs are not pure perl :)).

For more information about Perl's success, you can read Perl Success Stories.

If you find that scalability is an issue with your application, switching to mod_perl will likely remove the bottleneck. Further, because of the way that mod_perl is designed, mod_perl programs often run faster than their C counterparts. You also have the option of embedding C in Perl for very time-sensitive tasks.

Note that the Perl Success Stories link above has wonderful examples of how scalable Perl can be with the right team using it.

Regarding Perl vs. Java: one of my favorite quotes is by merlyn (pardon me if I am misquoting). He was facing an audience of students and there was some discussion of both Perl and Java. One of the students asked him how he dealt with the lack of strong typing in Perl. merlyn replied, "I just smile and move my programs into production before the Java programmer has his first compile."

Cheers,
Ovid

New address of my CGI Course.
Silence is Evil (feel free to copy and distribute widely - note copyright text)

Yes, I believe the human component of scalability is important, and increasingly so as cost of hardware goes down, and programming time/cost increases. Here's my own story that I think emphasizes your point: Three months ago I knew very little about Perl. I had heard of it, but that is about it. I have recently left a rather lucrative IT management job to get "back in the trenches" so to speak, after a 12 year hiatus from programming (I left off doing C programming in a Unix environment). My challenge was to develop an e-commerce web site for my brother-in-law for his business. I met with him (Pres/CEO) and his CIO, at which time the CIO told us that this project would take 9 months at least, and that is if I were a very experienced web programmer. (Their company, incidentally, uses MS tools/languages, etc.). At that time, I thought that MS was the way to go, and also their current company environment. After several weeks of studying HTML and various VB tools/books, I was about to explode in exasperation; I would spend days fighting with a number of MS’s “controls”, only to find out later that they had this or that “bug” in them. About to give up, I then turned to Perl, since I liked the looks of it (similar to my work with C), heard good things about it, and, of course it is free! Long story short, in 3 months I learned Perl (I’m not an expert, but picked up enough to program a relatively sophisticated e-commerce web site), learned Javascript, learned HTML, learned web-programming in general, installed/used MySQL (also new to me), and developed a relatively complex e-commerce site. I met with the CEO and CIO last week to demonstrate my software. Although I could tell the CIO was more than just a bit shocked that I had done what I had done in considerably less than his "best-case" 9 months, he still had the political wherewithal to say, (in front of the CEO) ”Oh, you used Perl, that’s not scalable you know”. So thank you for this thread, I am now armed to do a little verbal butt-kicking myself (at the right political moment of course!). Cheers, Jim Wilson

BTW: As part of the aforementioned episode, I did discover at least 1 downside to Perl; I finished the task in such short time that I put myself out of a job! Anybody know of any Perl shops that are hiring? Jim Wilson jrw@extremeco.net Have Perl Cookbook, Will Travel

I think for a lot of "corporate-type programers" Perl is intimidating, so they choose to believe the FUD.

The same features that make it so endearing to most of us here in the Monastery make it suspect for them:

• TMTOWTDI might be what we like about Perl, but it scares the Hell out of most of the World,
• Perl gives you Freedom, but with Freedom comes responsabilities, which sounds like "I'll be fired if I fail" to a lot of professional programers,

  I am sure we all agree that indeed there is a Poetry of Programming, but that sounds like anathema to a lot of Java-heads.

I could go on like this, but it mostly boils down to "I have learnt Java in school, why should I learn something else" and finally to "nobody has ever been fired for using Java".

Once they believe in this, then they have to justify this cultural reaction, and come up with tons of mostly-invalid-but-serious-sounding reasons.

BTW, Amazon.com runs on Perl, CNNSI uses it a lot, I had people from Boing attend a Perl class, and sure, Yahoo is moving away from Perl, but their plan is to use ... PHP! So scalability does not seem to be a problem for some rather huge organizations.

They use Perl for their backend data-processing and batch work. The PHP stuff is just for stitching together a bunch of include files at request time, and it's replacing an in-house system written in C.

Perl is not being used for large projects as much as PHP. Why? Because of a number of good reasons that many Perl advocates are ignoring:
1. mod_php is easier to install than mod_perl.
2. PHP is easier to learn than Perl.
3. PHP is fast.
4. PHP is powerful. You can now use PHP at the command-line. You can even use PHP for shell scripting or to build GUIs.
5. There are not as many ways of doing things in PHP as there are in Perl. This is a positive thing for PHP if you have large projects with a large team of programmers working on code for a long time.
6. If you need help with PHP, you go to one source: www.php.net. If you need help with Perl, you have many different websites offering different things (like documentation, modules, articles, etc).
7. PHP is open source, but better marketed than Perl.
8. PHP is buzzword compliant.
I have belonged to the Perl community for some time and I don't know if anyone else is noticing this, but the same websites are always referenced when discussing large projects. These sites are etoys, Amazon, CNNSI, Slashdot, Wired, TicketMaster, and few others. How long and impressive do you think the list of sites using Java and PHP are?

1. mod_php is easier to install than mod_perl.
   Well, I'll give you that one..
   
   
2. PHP is easier to learn than Perl.
   Sure, you trade power and expresiveness for simplicity. Which is why so many small projects use php and very few large projects do so.
   
   
3. PHP is fast.
   Mod_perl is fast, c is faster then both. ASM is probably faster then c.
   
   
4. PHP is powerful. You can now use PHP at the command-line. You can even use PHP for shell scripting or to build GUIs.
   Wow. I mean, wow man, so now php is atleast barely equivalent with every other language? Java is cmd line. Perl is cmd line. Python is cmd line. Same for guis.
   
   
5. There are not as many ways of doing things in PHP as there are in Perl. This is a positive thing for PHP if you have large projects with a large team of programmers working on code for a long time.
   See earlier arguements about making a single programmer more powerful vs a team.
   
   
6. If you need help with PHP, you go to one source: www.php.net. If you need help with Perl, you have many different websites offering different things (like documentation, modules, articles, etc).
   If you need documention for perl, you go to perldoc.com, or man. If you need to ask a person, you go to perlmonks.com/org
   
   
7. PHP is open source, but better marketed than Perl.
   Thats quite possibly true. Perl needs a better PR campaign.
   
   
8. PHP is buzzword compliant.
   Compared to what..?

Ask them this: if Java is so perfect, why don't the largest sites use it? Why don't Yahoo and Amazon run Java? Both of them do use Perl and C, but not Java.

You might also mention that TicketMaster/CitySearch runs on mod_perl.

An important question is how one positions Perl and Java in the programming language landscape.

Obviously, both are general purpose languages and each problem that can be solved in Perl can be solved in Java and vice versa... well, at least in some way, though the solution that's obvious in one language may be a nightmare to implement in the other.

As an example, just try and do some system administration tasks in Java: well, no one is even trying. Try to do some simple data transformations and manipulations in Java, it will only increase your love for Perl.

Perl has definitely made its mark in server side web programming, even before Java became a hype, let alone a hit. But here's an important point: Perl has traditionally been used for CGI programming, which -- at least without mod_perl -- requires a separate process for each incoming request. When using Java Servlets, this is not the case since those processes are instantiated at startup of the server. Moreover, servlets can be stateful, while this is a pain to achieve with CGI. So yes, for web programming Java has an edge for certain applications.

From a developer's point of view, both languages have their merits. Perl is flexible and allows for very fast development. Java is more strongly typed, has a cleaner syntax and is more disciplined, making it a better choice for "things that are to last" (although a lot depends on the programmer, it's easy to write ugly code in Java too).

The conclusion could -- and I think should -- be that there's such a thing as the right tool for the job, at least if the one handling the tool knows how to wield it.

What could you have done differently: well, emphasize that Perl is not limited to web programming, but is a very versatile tool for data munching and sysadmin, something that should be important in a course on UNIX.

Just my 2 cents, -gjb-

They all seemed to think that JAVA (insert silence) was God, and superior to all languages. Their main point was that java only spawned one process and could handle hundreds of users and that perl, on the other hand, could not.

Eh, this was a Unix Fundamentals class, wasn't? I've given Unix Fundamentals classes, and what we do it stuff like browsing the file system, looking at files, editing them, commands like cd, ls, more, cat, vi, grep, find. We talk about in/output redirection, pipes, exit status, awk, sed. A natural environment for perl.

What's the deal about processes? If I start a perl program, I start a single process. If I start a Java program, I start a single process. Now, they may claim a perl process is taking lots of memory, but so is your typical Java program. What they mean by "a Java program could handle hundreds of users" is beyond me. You aren't writing operating systems in your Unix Fundamentals class, I hope?

In my professional opinion, I find that Java *doesn't* scale. It's wasting too much memory - many applications come with their own Java run-time enviroment (because the "write once, run everywhere is a blatant lie"), and a handful of Java run-time environments quickly consume all the memory available, even if it's 2 Gb. Now, Perl doesn't scale well either, it's also memory hungry. With Perl it isn't so much the run-time environment (although that still isn't memory friendly), but the amount of memory data takes.

Mind me, I'm not saying Java is a bad language. It certainly does have its merits. It's syntax isn't complex, and not as rich as Perls. It has strong typing, and for a lot of programmers, Java is a better language than Perl. It keeps you further away from the idiosyncrasies of Unix. This is sometimes a good thing - but try to make a Java program that pays attention to environment variables. If I have to choose between Java and Perl, I can make that decision based on many criteria. For some criteria, Perl will win. For some, Java will win. But scalability isn't a criterium for which Java will be a clear winner (they both lose ;-)).

If something really has to scale, with all other things equal, C is the only logical choice.

Abigail

If something really has to scale, with all other things equal, C is the only logical choice.

lisp?

:-)

--- demerphq
my friends call me, usually because I'm late....

Perhaps not. Yahoo is replacing Lisp in it's Yahoo Store software for speed. (And I suspect to ease future coding.)
Paul Graham is "Mr. Lisp" so I assume the Lisp code was implemented very efficiently. There's still some Perl in the Store Manager though!.

-Lee

"To be civilized is to deny one's nature."

obviously a compiled language would run faster than a script

It's not obvious to me why this is the case. It's also not (quite) obvious to me why people think that Perl is interpreted and Java isn't.

Perl's biggest shortcoming is that it's so efficient, you can often get your job done before you stop to think about the "right way to do it".

Perl's scalability depends on how you do it. I'm sorry this answer isn't more meaningful, but it's a meaningless question. (Is a hammer scalable?)

Perl is only a love-hate relationship for people who don't yet grasp the subtleties that Real Programmers (to be elitist) encounter every day.

It's also not (quite) obvious to me why people think that Perl is interpreted and Java isn't.

It's obvious to me: because they have to run javac before they can execute Java code! As we know, Perl does this step implicitly, and Perl's compiler is so much more efficient than Java's that no one even knows it's there unless they bother to learn something about how Perl works internally.

The bottom line for anyone who doesn't already know this: both languages compile to bytecode and then interpret that bytecode. This is also how modern versions of PHP and Tcl work. So, explain to them that Java is no more compiled than PHP is, and see what they make of that.

Also, note that while Perl CGI has to do the compilation step every time, mod_perl and similar persistent technologies (and equivalent for PHP and Tcl) just keep the bytecode in memory after the first time it is compiled.

[People think that Perl is interpreted and Java isn't] because they have to run javac before they can execute Java code!

Many JVMs do incremental compilation, using technology pioneered in Smalltalk and Self, though it's probably true that the fact of there being a separate "compilation" phase, by a thing called a "compiler" confuses many people about what's really going on under the covers.

This should help clear up some of the misconceptions in the Java vs Perl argument...

-- Can't never could do anything, so give me and inch, I'll make it a mile.

They're spouting nonsense, probably based on simple ignorance. I've worked on a Perl-based application server that could handle thousands of HTTP requests per minute, corresponding to several hundred logical sessions. We were limited by the network pipe more than we were by CPU.

The advantage you get over quicker time-to-market can offset quite a bit of scalability for many applications.

Michael

Their main point was that java only spawned one process . . .

Last time I checked the output of 'ps -A' with a JVM running, Java spawned a lot more than one process. It could just be my Virtual Machine (IBM on GNU/Linux), and I suspect those extra processes are for the JVM, not the actual program. Java might have some superiority with threads, but this is probably a temperary issue as Perl threads continue to get better.

They're just talking about web programming with servlets, not normal programs started with java SomeClass. With servlets it's true that each request goes to a thread and there is only one process (although many servlet containers can run multiple processes to take better advantage of multi-CPU systems).

Also, I remember I used to see separate entries for each thread in top on Linux. Not sure if this is still the case or if ps does the same thing.

For those of you who wonder why your JVM in Linux spawns so many processes:

"Unlike some other implementations of Unix (such as AIX or Solaris), the Linux kernel itself does not provide an abstraction of allowing several kernel threads to be packaged inside a process. Instead, a special version of the fork() system call (also known as clone()) has been implemented that allows you to specify that the child process should execute in the same address space as its parent. This system call is then used by the Linux threading library to present a thread programming abstraction to the user. From the user interface standpoint, practically all of the POSIX thread interfaces are fully supported under Linux; however, the implementation is not fully POSIX-compliant due to some problems in signaling. The result, however, is that each Java thread in the IBM Java VM for Linux is implemented by a corresponding user process whose process id and status can be displayed using the Linux ps command."

http://www-106.ibm.com/developerworks/library/java2/

Updated 2002-12-10 by mirod: turned the URL into a real link

Performance is often misunderstood... most frequently performance issues come from 2 sources... (a) database access, and (b) file system access (like writing logs). What's important to note... neither one of these is related to a particular language. Caching and data pre-preparation go a long way to resolving these issue.

Keeping processes resident in memory also can be a big improver of performance... you aren't going to get a clean comparison of JSP to straight CGI if you aren't using mod_perl or something similar.

Perl does a lot of jobs well; I think it is much more suited to 'batch' type processing than is java... for example, grab some data out of a database, drop it into an XML file. You then pass that file through a XSLT filter(s); get out some pregenerated HTML, marked up with a bit of javascript. That tends to be seriously scalable... and easy to support and debug. 'Use the right tools for the job at hand' is what I would have told them; sometimes that tool is Java and JSP... but often its not.