OP here.

We're using the Google Cloud Platform. They have tons of docs on uploading files, but the basic variants are all explained here, with variants under each tab: gsutil is their commandline tool, Code Samples shows APIs for various languages (Python, Ruby, Go, Java, Node.js, etc.), and REST APIs shows that.

We haven't yet run detailed profiling. Our original system simply copied files to a remote filesystem, and thus could use regular filesystem tools. It was vastly faster, even though the raw network speed isn't appreciably different.

I appreciate that some kind of system to batch the files would be better. The problem is that the underlying code is extremely complicated: merely selecting the files for uploading takes hours, and every upload requires a number of database updates (which ideally should take place at the same time as the upload, i.e. the location recorded in the database should match the location of the file, so selecting the files all together and then uploading them in one batch will be a problem).

Using the REST APIs would require a lot of manual housekeeping (checking for successful completion, retrying interrupted uploads, etc.), that is handled automatically by gsutil and the language APIs, which is why we felt that it was an advantage to use one of these tools. Perhaps we should just do this, but I'd hoped that some way of using one of these tools could still be more stable than rolling it ourselves with the REST API.

In reply to Re: Overhead of Inline::Python? by Anonymous Monk
in thread Overhead of Inline::Python? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.