The "parsemedline.pl" code that you cited (at the biotext.berkeley.edu web site) could have been
a lot shorter (with no loss of intelligibility or maintainability), if the code authors had made more thoughtful use of perl data structures (HoH, HoA, HoHoH, and so on), instead of declaring vast numbers of simple arrays with long names. (Personally, I think shorter code is easier to maintain; and declaring arrays to keep track of the names of hash keys is a lot easier than keeping lots of differently-named arrays.)
As for run-time efficiency compared to a java implementation, I don't know how the java version handles RDBMS insertions, but the perl version cited here is obviously working at a serious disadvantage. The code is doing two things that I would normally call bad ideas:
- It is doing a presumably large number of inserts via DBI, instead of using a native flat-file loader facility that comes with virtually every RDBMS. If you simply convert XML data to a flat file and feed that to something like "mysqlimport" or oracle's "sqlload", the database will be loaded in a small fraction of the time that DBI would take doing "insert into..." statements.
- To make matters worse, for each insert statement, this code is preparing a new statement with quoted values, executing, and then calling "finish" on the statement. If it simply stored a set of prepared statements, using the "?" placeholders for values to be inserted, run-time would be noticeably faster (though still slow compared to an RDBMS native text-file-import tool).
One other nit-pick: the commentary in the code is quite good as documentation, but it would be better as POD (and this would be so easy -- there's no good reason not to do so).
I counted over 3700 lines (excluding blanks and comments) in "parsemedline.pl"; I don't know whether I'd try to boil it down (not sure I want to pull in 40+GB of data from a field I know nothing about), but as a rough estimate, I'd guess this could be done, using appropriate data structures and loops, with well under 1000 lines. Hard to say what sort of speed differences would result, but if there is ever any "evolution" in the XML data format, a shorter version of the code would be a lot easier to fix, I think.
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.