You'd be wise to completely refactor the code and to separate the logic for parsing and processing.

ATM this is a maintenance nightmare (sorry)

parser

This "format" seems to have a strict grammar:

• lines have <indentation> <capitalized KEYWORD > <datafields>
• the <datafields> are either space separated or {grouped by curlies}
• if lines are nested the level is finished by END_keyword

So write a _generic_ parser (and better implicitly check for correctness) by

• parsing each line with a regex,
• keeping record of nesting levels
• putting parsed data into a hash of hashes

If this parser works correctly you can start measuring the _speed_ for your huge file.

Now you are able to tell if the DB operations are the bottleneck (most likely they are)

processing

You should try to bundle many DB-operations, cause there is always a overhead.

If the complete HoH after parsing the whole file is to large, you must define "rules" which hook into the parsing.

This can be done by checking after each line for suitable callbacks to evaluate your temporary data so far.

Define a callback-hash $action{KEYWORD} holding subs to be called if a keyword is processed.

HTH! :)

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.