I appeal to the Monks to aid me in my quest for true Laziness...

My employer, an ISP, uses MRTG everywhere. A while ago, I was given the task of consolidating all the one-off MRTG servers into one unified system, and so far, I've been able to pull it off. The systems now consists of two pollers and one web server. The web server mounts the poller's data over NFS and each night copies over all the MRTG config files from each poller. Here's where it gets hairy...

Each night a perl script I wrote that uses the readcfg() and cfgcheck() subroutines from MRTG_Lib to load and parse all of the MRTG config files into one giant, complex hash in memory. I then do some post-processing on this hash before writing it to disk by using DBM::Deep. - I call this part the Compiler.

The website's index pages (one for each node, plus one 'Master' page with links to the node indices) are re-generated from the data in the DBM::Deep DB hourly to show changes in utilization (queried on-the-fly from the RRD files) - I call this the Indexer

The actual target detail pages are displayed using 14all.cgi (Which reads the RRD files over the NFS mounts)


Anyhow... While my pollers are keeping up with polling the ~450 devices between them, the web server's Compiler is now choking on the sheer number of MRTG config files. In particular, the Compiler hits about 250MB of RAM usage and then stays there forever, never finishing the job.

It's getting stuck in the readcfg() routine from MRTG_Lib 
    print "Reading MRTG config from $main_mrtg_config...\n"; # DEBUG
    readcfg($main_mrtg_config,\@device,\%cfg,\%tgtcfg);

    print "Checking MRTG Config...\n"; # DEBUG
    cfgcheck(\@device, \%cfg, \%tgtcfg, \@target);

[download]
I mentioned earlier that I'm using DBM::Deep to store the final product... that was an add-on I made to my own code when it became clear that parsing the configs every hour was impossible, so I broke the Indexer in two, thus creating the Compiler.

I've tried tie-ing the %tgtcfg hash that readcfg() builds to another DBM::Deep DB but all it does is take an eternity at 100% CPU usage. (I killed it after 12 hours... and the file had grown to over 300MB)

I've tried using MLDBM with BerkeleyDB::Hash and Storable, but it just crashes with this error: 
ERROR: CFG Error in "setenv[192.168.52.1_1]", line 18:  must be XY="dd
+dd" AASD="
kjlkj" ...

[download]
(which is thrown by the code in MRTG_Lib)


I'm at wit's end (it wasn't a long way)... is there any magic I can work without having to completely re-write my Compiler?

In reply to Parsing a really, REALLY big MRTG config by Hercynium

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.