Just to throw my two pennies into the mix, I am not surprised that either forking for every packet or implementing the disk queue was slow. Both forking and disk I/O have high overheads. I really think your initial solution was a pretty fair idea.

If this is still too slow, I might suggest a three-process approach. The packet catcher will spawn a child every 1000 packets. The child will spit the packets to disk while the catcher gets back to work. A third process watches for new files to be created ( maybe naming each file with the child's PID so you would know when it exited and the file was complete ) and does the parsing. This would gain some speed since the third process could keep a permanent connection to the database - DBI->connect is slow. It also makes this almost nightmarishly complex.

Just brainstorming now. What if you were to use ( since I just offered an answer using these ) one of the IPC shared memory modules? Using some kind of ring buffer in the shared memory, this would allow the parent and child to work asynchronously. It would also elminate some significant overhead, as the child could hold a more or less permanent connection open to the database.

mikfire

In reply to Re: Perl Performance Question by mikfire
in thread How can I maximize performance for my router netflow analyzer? (was: Perl Performance Question) by IkomaAndy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.