Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Server side computational suggestion

by sherab (Scribe)
on Mar 09, 2010 at 18:59 UTC ( [id://827607]=perlquestion: print w/replies, xml ) Need Help??

sherab has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks! I have been a perl developer for some time. Needless to say I use it every chance I get. What I am tasked with now is doing some hard core computational finance related tasks. I don't know how much many of you know about computing stock related data but the types of stats I need to come up with are numerous and these would have to be computed every day for numerous stocks and sometimes on the fly. It's massive number crunching. (It will need to be done everyday for charts, etc)

I am all about the best tool for the job but I'm not sure that Perl represents my fastest choice with this kind of need. It may be the ONLY time in my life I have said that about, what I feel, is the best language ever.

Does anyone know if Perl would indeed be worth using for this gargantuan number crunching task? I absolutely refuse to remove Perl from the equation. The site I am working on needs things like authentication and I have my mind made up about using Catalyst. I would think the only other way to handle it would be to still use Perl but to hand off the extremely heavy lifting to some C++.

Any words of advice or input is appreciated.There is no other group's opinion I respect more than you monks.

Replies are listed 'Best First'.
Re: Server side computational suggestion
by tirwhan (Abbot) on Mar 09, 2010 at 19:24 UTC

    Try it out. Implement the entire site in Perl, then find out which parts are taking too long. Then see whether you can find a faster Perl solution (maybe PDL is suitable for your problem space) or whether you need to reimplement parts in another language. Embedding C finctions into a Perl module is documented well enough that it should be easily possible to change the number-crunching aspects of even a complex app from one to the other.

    This gives you a working (albeit possibly too slow) solution in the minimum period of time, and you can improve the parts that need improving as you go. At worst (if for some unlikely reason it turns out you need to do the entire site in C or C++), the initial Perl solution is a good prototype. You always learn when you do something for the first time, and Perl is well suited for rapid prototyping.


    All dogma is stupid.

      Just to amplify on what tirwhan said: Get the job done before making it faster. Often, you'll find that it'll be fast enough. When it isn't, you'll often find that:

      1. Adding CPU/RAM/DISK/whatever may be cheaper than coding time.
      2. Improving the algorithm may be enough to make the system fast enough without recoding in a new language.
      3. Your processing time may be dominated by I/O time, making the language you choose irrelevent.

      Back in the old days, I often hand-optimized code in assembler or C to make it fast enough. Now I rarely find the need. I frequently do some serious processing on fairly large files, and the processing time is small compared to the I/O time. So it would be faster in my case to get faster boxes of disks rather than spend any time on updating algorithms, translation, etc.

      ...roboticus

Re: Server side computational suggestion
by almut (Canon) on Mar 09, 2010 at 19:19 UTC

    Maybe PDL is what you're looking for (all the relevant parts are implemented in C/XS).

    Using pure Perl for number crunching is probably not a good idea (speed-wise).

Re: Server side computational suggestion
by bot403 (Beadle) on Mar 09, 2010 at 21:21 UTC
    Although PDL is probably a better solution Inline C could work if you need to write some custom low level things PDL doesn't provide. This way you get the speed for critical sections and all your glue and higher level work can be in Perl.
Re: Server side computational suggestion
by girarde (Hermit) on Mar 09, 2010 at 23:40 UTC
    Forth would be scary fast for this. Aside from the time to install and learn forth.
Re: Server side computational suggestion
by ahmad (Hermit) on Mar 10, 2010 at 00:09 UTC

    I cannot imagine what you are talking about here.

    What kind of operations you'll be doing with these numbers that could take too much to get completed ?

    Are those data preloaded into a database? or you'll be parsing some text files over and over to do the calculations?

    Have you considered relying on the database engine (if you use one) to do the basic calculations to make things faster/smoother.

    If you're going to make it in Perl then I'd like to get updated on this (Might give us an estimation on how far we could go with our Perl based solutions :P ).

Re: Server side computational suggestion
by JavaFan (Canon) on Mar 10, 2010 at 14:18 UTC
    It's massive number crunching.
    That's what God gave us FORTRAN for.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://827607]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (2)
As of 2024-04-20 05:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found