jasonjohn has asked for the wisdom of the Perl Monks concerning the following question:

I'm considering implementing a multicasting chat server in Perl, as outlined in the book "Network Programming with Perl" by Lincoln D. Stein

However (assuming it's a success) I'm expecting around 1000 simultaneous users.

Given this bandwidth, I doubt Perl is suitable? The Perl application would have to run as a dameon/service and therefore faster? In any case, the server would have to be considerably beefy. Any ideas on how much (ball-park) bandwdth a 1000 user chat would generate?

I'm considering using Stein's book as a learning tool and then rewriting the code in C/C++. Or maybe wrap Perl around a C/C++ binary for the speed-critical parts... but then why use Perl at all?

A long road a head, me thinks ...

Any links or suggestions would be greatly appreciated :o)

Replies are listed 'Best First'.
Re: Perl chat server?
by cjf (Parson) on Apr 28, 2002 at 03:28 UTC

    merlyn wrote an column on writing a chat server (in under 100 lines nonetheless) that may be of interest.

    As a slightly relevant side note bound to draw a flame, Java Network Programming is an excellent book if you decide Perl is not suitable for the task (not that I approve of Java ;).

[reply]
Re: Perl chat server?
by chromatic (Archbishop) on Apr 28, 2002 at 04:15 UTC
    IRC does sound like an existing wheel.

    If I were you, I wouldn't worry about supporting a thousand users right away, just like I wouldn't worry about handling 50 hits per second on a new web site. Since this is a learning tool, do it in the language you know best (or want to learn most). Do it correctly, and then worry about doing it quickly. I'd be very surprised if Perl couldn't handle that bandwidth, given how much mod_perl can serve on a moderate box.

    You might also find POE useful.

[reply]
Re: Perl chat server?
by Saveth (Sexton) on Apr 28, 2002 at 03:50 UTC
    To answer your question, the average traffic on one of my public IRC servers with a 150 user average is 100B/s. So, I'd guess 1000 users would use somwhere from 650-700B/s, which translates into about 55M/day, or 1.65G/month.
[reply]
Re: Perl chat server?
by mattr (Curate) on Apr 28, 2002 at 11:24 UTC
    This is a great question, and not academic for me. Thanks. I'm working on a system which needs to run a 3000-user chat (through web input forms or maybe irc clients), do something with the information that is chatted, and serve web pages at a lower rate. Some parts of it would resemble IRC except that some other interesting things will be happening.

    I just read some parts of the Stem Systems site and it is very interesting! Though I was thinking an apache mod_perl server, or possibly a mod_perl server that logged into an irc server as a bot, might be the ticket. My own application will not be injecting messages into the chat stream, but processing them, and certainly there has to be a bridge from the live-message-stream paradigm to the apache hybrid page server paradigm. Maybe stem would be overkill, or maybe it could pay for itself in a couple of development cycles. Can't tell yet.

    But while Stem so far emphasizes system administration tasks, it might also be the answer to the P2EE folks' work.. at once relevant to a lot of things Perl people do now, instead of what people think we should be doing in the future. It has config files, messaging, and a cookbook of networking tools. Has anyone used this? It seems very cool that you can spread your objects across a number of servers, I guess failover is the next thing to sprout? (sorry.) Stem does not yet guarantee message delivery which could maybe be a problem if used as an IRC client.

    In comparison, POE which seems to be used for lots of IRC bots is apparently good enough to run stock exchange operations and handles high snort load with loghog. I guess the choice is between serving all chat from a slower main program so you don't miss anything, or sync with a stream and hope you don't get kicked or overloaded.

    One thing though, not that I expect it would be a problem considering the people involved, but you have to jump through some hoops to get it. In fact I have just stopped myself from downloading it because I don't want to have to fill out the very long questionnaire twice, once now and once when I get back to my office in some days.

    I figure when they say it will be GPL by 0.06 this might have to do with hammering things down for Perl 6..? But it says only free for noncommercial use and I'm not sure how that would work. Perhaps this means I would owe Stem money if I get paid? That isn't GPL, kind of shades o' gray. Any ideas merlyn (an advisor to Stem Systems) ?

    I think it could be quite hot to have a high power IRC-compatible module in Stem, the sample code of which has objects responding to tty messages. Wondering about performance and security.. humm better start reading.

    Update: POE::Wheel::FollowTail might be the answer. Run a proven bulletproof IRC server, and use this module to feed POE with the chat log line by line as it grows. Or maybe do the same thing without POE, erm. More studying..

[reply]
      Why not just use, and extend, existing bots written in perl or write your own using a framework like Net::IRC ? I use Net::IRC to power a bot I wrote. Its a verry nice event driven framework for connecting to IRC and handles all of the IRC commands for you.

      I know some pepole that really like perlbot: http://perlbot.sourceforge.net Tho I haven't used it it looks to be a very nice project..
[reply]
        Thanks for the suggestion. I'm looking at Net::IRC bots, but in my case I cannot afford to miss a single post, so I won't be able to say for sure until I test. Am also worried in general about security/administration of IRC and hardware resources needed.. so am looking for how others have done large scale systems (even slash or a normal bbs) not just IRC which has more features than I need or want.

        Will check out perlbot and the other links below. --Matt

[reply]

      A quick note about POE and chat servers: The distribution includes a very limited one in its samples directory.

      Addendum: I have since revised the sample chat server and placed a copy of it on the web.

[reply]
        Yes Net::IRC has lead me to POE which has lead to much more reading on my part about event models. POE looks really awsome.. and could be helpful on this project. POE on CPAN
[reply]
Re: Perl chat server?
by Felonious (Chaplain) on Apr 28, 2002 at 04:28 UTC
    I recommend implementing a pure perl version first and testing it. You could always move speed critical portions into an XS module if needed. I think you'll be pleasantly suprised by how well perl handles the load though with a little tweaking. I once wrote a toy (minimum functionality) web server that could out-perform apache (with keep-alive turned off... I abandoned my script before getting keep-alive working in it).

    -- O thievish Night, Why should'st thou, but for some felonious end, In thy dark lantern thus close up the stars? --Milton
[reply]
Re: Perl chat server?
by uri (Acolyte) on Apr 28, 2002 at 05:59 UTC
    Stem already has a prototype chat server as one of its demos. you can write a chat command parser and a control module and quickly develop your own design. it should easily handle 1000 users. look at stemsystems.com
[reply]
Re: Perl chat server?
by Saveth (Sexton) on Apr 28, 2002 at 03:32 UTC
    If you're going to do it in C/C++, why even start with Perl? Some elements of the languages are similar, but, by far, they are not the same language. I'm sure there are plenty of resources on chat servers written in C/C++.

    Why are you reinventing the wheel, anyway? Isn't IRC suitable for your purposes? Your description of the dilemma was a little too vague, here.
[reply]
Re: Perl chat server?
by abstracts (Hermit) on Apr 28, 2002 at 08:31 UTC
    One more thought that no one touched upon: Should one fork a 1000 processes to handle 1000 users, or should one use select on 1000 sockets in a single thread? Or a combination of the two? My gut feeling is that for 1000 users, one should fork around 30 processes, each handling 33 users or so. Would make a nice experiment when I'm done with finals :-)
[reply]
Re: Perl chat server?
by Seumas (Curate) on Apr 28, 2002 at 18:41 UTC
    I'd suggest looking at geektalkd which is a perl version of chatd. It has a very small footprint (19k) and clients are simple to write for it (the simplest of which doesn't need to be more than a couple hundred bytes).
[reply]
Re: Perl chat server?
by strat (Canon) on Apr 29, 2002 at 11:47 UTC
    If you want to use IO::Select, maybe http://hotwired.lycos.com/webmonkey/97/18/index2a.html could give you some ideas on how to do it. I played around a lot with the base presented on this page, and enhanced it and the like, but I can't tell you if this can handle thousands of connections because I've never tried more than about 20 connections at the same time.

    Best regards,
    perl -le "s==*F=e=>y~\*martinF~stronat~=>s~[^\w]~~g=>chop,print"

[reply]
Embed Perl
by scottp (Initiate) on Apr 30, 2002 at 06:58 UTC
    If you do consider writing your code in C/C++, and there is no bad there if you require it, then pleaes consider embedding Perl. This may sound silly, but so many daemons are rewritten in Perl just for all of the perl bindings when it could be done by embedding perl into an existing daemon.

    I use Apache this way (of course that is obvious) but also OpenLDAP, UIRD (Universal Infrared Daemon) and more.

    Scott

[reply]
Re: Perl chat server & Client
by Anonymous Monk on Apr 29, 2002 at 20:43 UTC
    You may want to checkout Perl chat on sourceforge. It's a good place to start if you're writing a Perl chat server and client. It can be found at http://perlchat.sourceforge.net
[reply]

Back to Seekers of Perl Wisdom