Hi,

I'm designing a algorithmic trading app and would appreciate any wisdom that you would be so kind as to bless upon me.

I started with the following design:

main program, using IPC::Shareable create and tie %cache.

Run a sub dataCollector in a new fork, which loops every 0.1s incrementing $i, and then adds the data to a hash giving a {$marketId}{$i}{$broker}{price}. Each time a new set of data is published send a SIGUSR1 to the other processors to alert them that they have new work to do. Once a market is out of it's window of interest approximately 30 minutes later delete $cache{marketId}. Each 0.1s loop yields about 6k of data, and there may be upto 8 markets that are of interest simultaneously to total memory requirement is about 420Mb.

Main program will call subs for each strategy, which will fork and return the forked pid to to main. The strategy workers will receive their interrupts and handle them by reading the shared memory, do various computations to decide if the conditions are good to trade or not.

The dataCollector is the only process that would need to write to the cache, and each worker would be triggered simultaneously and be reading the same piece of memory. Another worker task will be writing the data out to file for back testing strategies, however if data is lost it's not an issue.

Hope that makes sense in what my objective is. What I would appreciate your wisdom on is:

1. IPC::Shareable doesn't seem to handle deeply nested hash refs very well and probably therefore isn't the right choice. Is there an alternative that I should consider?

2. Most of my reading has pointed me towards using an in-memory database. In your experience which free database would give me the fasted throughput in this scenario?

Please turn your flame throwers up to max and burn this down, and help me get to the right foundations.

Thanks

Zapoi

In reply to Shared Memory Cache or Database? by zapoi

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.