gb_lexter has asked for the wisdom of the Perl Monks concerning the following question:

I am in a state of confusion and hope to find some enlightment.

We are looking for the right language to use in our next project. It will be the control of a large PostgreSQL database (>3 million entries, including several fields (also large textfiles in each)), make here excessive DB-searches, using the browser as a front end. We want also to make automatic research on other online-dabases via the browser and to update our DB with XML files. The usage of unicode is required.

I myself have some experience in C, LabView and a little PHP. I was told now to evaluate which language would be best for our project (suggested were PHP, Perl, Python and Java), but I find it hard to find informations about it, since it seems to me, that there is a battle of believe on the net about these languages, blurring the real issue a bit. Nevertheless, I hope to get some some advice from you concerning my problem.

Thanks for your time,

Gunnar
  • Comment on Advice of picking the language for my job, please

Replies are listed 'Best First'.
Re: Advice of picking the language for my job, please
by samtregar (Abbot) on Feb 04, 2005 at 16:47 UTC
    It sounds like Perl would be a great fit for your job. Perl is a great choice for building database-driven websites and has good support for XML and Unicode.

    Since you don't know it yet you'll need to give yourself time to get up to speed. I suggest you pick up a copy of "Learning Perl" from O'Reilly and find some small jobs to do while you learn. When you're done with that proceed to "CGI Progamming with Perl," "Programming the Perl DBI" and "Network Programming with Perl." These last three contain critical information you'll need to build your application using the best Perl techniques.

    -sam

Re: Advice of picking the language for my job, please
by FitTrend (Pilgrim) on Feb 04, 2005 at 17:21 UTC

    We faced a similar decision in 1999. Our company made a strategic choice to write all products in perl because we wanted to compete by providing a 100% web-based solution.

    We use Apache, ActiveState Perl, and MySQL 4.0 mostly for all our products today. Not to advertise, but you can read about what they do on my company's website or on my personal website for a little more technical description.

    We found that most web-based tasks can be quick to respond and provide users with an easy to use interface written in Perl, HTML/CSS, DHTML and JavaScript is more than adequate. We collect and store all kinds of data for small to enterprise size organizations (Network Inventory, SNMP polling, Desktop Inventory, Syslogs/eventlogs/snmp traps, and trouble ticketing)

    With that said, the volume of data can present some problems when the back-end is written in perl. Over time we have overcome some complicated issues such as polling 1000s of reports in 5 minutes, receive 1000s of syslog events, and operate a call center within 1 server which essentially is constantly reading/writing to the MySQL back-end.

    We did play with postgres a few places along the way and found the perl interface to postgres to be less then desireable. I can't say if it has been improved or not since I looked at it last (Late 2002).

    Based on our success, I can say that it can certainly be done in Perl. However, depending on what you are doing, its important to make sure (probably are) the appropriate modules out there before coding the project. Modules can save time and make you more nimble to add and enhance features.

    Hope this helps.
    Marc

      We did play with postgres a few places along the way and found the perl interface to postgres to be less then desireable.

      Are you referring to DBD::Pg or PL/Perl? If the latter, it's been overhauled in the recently released PostgreSQL 8.0. If the former can you say whether PHP's native PostgreSQL functions are likely to be better?

        As I recall, we were using DBD:Pg. I can't say whether the PHP stuff works since we've only recently (within the last year) began including some PHP tools as part of our solutions. However, I appreciate the update and will definitely keep PL/Perl in mind when we revisit this! -Marc
Re: Advice of picking the language for my job, please
by talexb (Chancellor) on Feb 04, 2005 at 16:49 UTC

    I have written two nodes about the language of choice, and the second article sounds like your situation: a fairly straightforward application, so that suggests PHP.

    Perl can be a hard language to learn. For me, it was easy, but with 15 years experience in coding in C and a decent grounding in Systems Design, I had a head start. And I'm still learning new stuff, like How to structure tests that span several modules, six plus years after discovering Perl.

    Perl is a lot of fun. It depends I guess on how much of an intellectual challenge you or your company are up for. And only you can answer that question.

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

    Update:Fixed typo 'gues son' -> 'guess on'.

Re: Advice of picking the language for my job, please
by Anonymous Monk on Feb 04, 2005 at 16:30 UTC
    If I have to pick a suitable language for a project, a number of factors are of influence. But by far the most important factor is "in which language do the programmers involved feel most comfortable".

    Another consideration is having to do a specific task for which a particular language has excellent tools, and others don't. But since your job seems to revolve around databases, it doesn't play much of a role. All important modern languages have excellent database interfaces to all major databases.

      I agree with Anonymous Monk. It really is in which language do the programmers feel most comfortable with. If they're comfortable with all then let them pick.

      I'm sure I'm going to get downvoted for saying this, but my prefered language for any kind of DB to web work is PHP. Perl I prefer for most any other back end work. I'm just more comfortable with doing that stuff in PHP (and PHP's syntax for this stuff seems easier to me).

      IMHO either will handle the task. The language is also not the only factor to consider. The database and the webserver should also be evaluated as well as how well they all interact. I haven't used PostgreSQL (mainly b/c we're stuck with MySQL and Oracle) so I can't comment on how well it plays with Perl or any other language.

      Some semi-interesting links:
      http://www.linuxgazette.com/issue67/nielsen.html
      http://www.linuxgazette.com/issue69/nielsen.html

        > It really is in which language > do the programmers feel most > comfortable with.

        I do not agree here. Certain languages have certain advantages. A lot of programmers have a feeling about different languages. There are only a few studies about the effectiveness of different languages.
        A good starting point to understand the differences is an article available online:
        Lutz Prechelt. An empirical comparison of seven programming languages. IEEE Computer 33(10):23-29, October 2000. (http://page.mi.fu-berlin.de/~prechelt/Biblio/jccpprt_computer2000.pdf)
        Regarding DB, you should also reflect some performance issues. There was a section "how fast is the DBI?" in the CPAN documentation. I cannot find it anymore. Must have been removed.
        But I'd say it is not DBI itself you should point your finger if you have performance issues.
Re: Advice of picking the language for my job, please
by samizdat (Vicar) on Feb 04, 2005 at 18:41 UTC
    I've used PHP for most of my web database work, but my new task set has me using Perl a lot more again, so I've got some experience to comment with.

    I can't address Java, because I've never used it, although one of my programmers swears by it. Ditto Python.

    Perl (IMHO) is syntactically harder than PHP. Expect to have a learning curve here, and be prepared to buy a copy of every o'Reilly book released on Perl. That's the only way I survive Perl, and, even at that, I need Perl Monks troops to come to my rescue occasionally. Perl is (IMHO) more powerful than PHP, especially when you compare the module archives available for each.

    I would agree wth the poster who argued for writing your abstraction layers in Perl as opposed to C. I used to wing it on projects with different languages for different parts of the job, but, over the long term, maintenance will bite you. This is going to be a big, long-term project, and I'd evaluate it carefully in that context.
Re: Advice of picking the language for my job, please
by FitTrend (Pilgrim) on Feb 05, 2005 at 21:18 UTC
    The only other advice and I can give is regardless of what language you use, its equally important to make sure you structure your schema to maximize performance.
      And also don't forget to index any columns that might benefit from it (basically anything used as a key). In the past I built a complete site with Apache 1.3, mod_perl, template toolkit and PostgreSQL (using DBI to interface). It was fast, but only after adding a reverse-proxy in front of the mod_perl server. The only downside was the tremendous amount of memory this setup used. Actually there was another downside as well: it took me some time to figure out the intricacies of mod_perl. If you don't even know Perl to begin with, then the learning curve will be much higher for you. There are alternatives to mod_perl (eg, SpeedyCGI) but I'm not familiar with them and so can't comment on that. Needless to say though, plain old CGI scripts can only go so far...
        In case anyone was wondering, like me, what a reverse proxy is, or why implementing one would achieve dramatic performance improvements, here's a Definition of reverse proxy:

        "Reverse Proxies

        A reverse proxy is a gateway for servers, and enables one web server to provide content from another transparently. As with a standard proxy, a reverse proxy may serve to improve performance of the web by caching; this is a simple way to mirror a website. But the most common reason to run a reverse proxy is to enable controlled access from the Web at large to servers behind a firewall."

Re: Advice of picking the language for my job, please
by Grundle (Scribe) on Feb 04, 2005 at 17:00 UTC
    In my experience I would say that you should NOT go with PERL to do any of your DB work. I am doing exactly what you want to do (Extracting XML and building XML files and inserting them in to the DB). PERL has an especially difficult time interfacing with PostGRES and there are currently some bugs and efficiency issues in the existing CPAN modules that might allow you to do this.

    My suggestion is that you use C to make a database abstraction layer, where all your logic and database calls are made through C. All your server-side programming can be done through PERL (because perl is excellent when it comes to working with verbose data) making function calls to your C API. You should never have to write any SQL in the perl (just DB function calls defined in your C).

    This has proven to be a very efficient and ultimately time-saving process. It has also helped in debugging and optimizing our DB performance.
      I might suggest that the fact that you made a database abstraction layer at all led to your benefits rather than the choice of C for that layer.

      I suspect you would have realized similar benefits had you written the abstraction layer in Perl.

        Like I said before, we have tried this before and Perl has some serious efficiency issues with respect to PostGres. When you start getting large data sets back it will simply crash. Also since we noticed that the current PostGres CPAN modules were buggy we decided that instead of re-writing them in PERL we would just go ahead and do it in C. Of course it all boils down to design preferences and what the Developers are comfortable with.
      Perl exists as a built-in tool to postgreSQL, it's called pl/perl. See PL/Perl documentation from postgres website
      It's simply that using DBI to interface postgres to perl is NOT the best way; but perl integrates extremely tightly and efficiently to postgres.
        I've been thinking of using Perl with PostgreSQL recently for a proejct. Can you tell me why DBD::Pg is any less efficient than DBD::mysql?
Re: Advice of picking the language for my job, please
by nmerriweather (Friar) on Feb 07, 2005 at 22:51 UTC
    I would ask the following: What is the development timeline of this project? Are there many users of this web project, or just a few that do intensive things? If you have a short timeline to develop, or expect few users (but the ones who do are doing intensive operations), then I think python would be the way to go. Its blazingly fast to develop and prototype, but as a web-language, its not very efficient (unless you forgo running it in apache and run the app as a twisted-python server) If you're looking at a longer timline to develop, then I'd look at perl and php. I think mod_perl is a great environment for building large complex web apps. I think php is great for doing highly user based stuff. The way php files are written/executed though, I'm not a large fan of it for "large" sites in regards to the number of pages one can navigate to. If you have lets say 10-20 possible page views/page types, I think php works well at even very high volume use. Right now I'm working on a project that has many many many variations on pagetypes, and I dont think it would have worked well on php at all. In oop mod_perl though, it is very natural and nice.