Good Day Monks,

I am getting ready to write a Java Applet that will suck data from a many local database servers and publish it to a central Perl/CGI content management system. This publishing process won't be done more than a couple of times a month per server, but each time will involve the transmission of well over half of a GB of data.

I am trying to decide The Best Way ™ to communicate the database data to the server. Here are the options I have come up with:

- Just post all of the data to a CGI in one request. I have never attempted to post this much data in a single HTTP request. Can it even be done?

- Have the applet post individual records at a time. Since it would be posting to a CGI each time (upwards of a million requests), this would cause a lot of overhead on the server.

- Have the applet post individual records to a mod_perl script. This seems like the best HTTP based option so far. There would be a small amount of data on each request, and the resident state of mod_perl would avoid most of the overhead involved in the second option (but there still just less than a million requests).

- Don't use HTTP, open a single socket to some server process and do all of the communication through that. This would be the most flexible and powerful option, and would probably be the most efficient. For example, as the applet is writing data to the socket, the server could be reading and processing it at the same time. But this also entails a lot more work.

I definitely won't need an RPC framework (like RMI, CORBA or SOAP). That would be overkill for the kinds of processing I will be doing.

I also have the option of submitting several records in each request. In general, I have complete control over both ends of this.

So, has anyone else had to do something like this? What have you tried? How well has it worked for you?

Thanks for your time,

Ted

Ted Young

($$<<$$=>$$<=>$$<=$$>>$$) always returns 1. :-)

In reply to Submitting large data to CGI by TedYoung

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.