Hercynium has asked for the wisdom of the Perl Monks concerning the following question:

I appeal to the Monks to aid me in my quest for true Laziness...

My employer, an ISP, uses MRTG everywhere. A while ago, I was given the task of consolidating all the one-off MRTG servers into one unified system, and so far, I've been able to pull it off. The systems now consists of two pollers and one web server. The web server mounts the poller's data over NFS and each night copies over all the MRTG config files from each poller. Here's where it gets hairy...

Each night a perl script I wrote that uses the readcfg() and cfgcheck() subroutines from MRTG_Lib to load and parse all of the MRTG config files into one giant, complex hash in memory. I then do some post-processing on this hash before writing it to disk by using DBM::Deep. - I call this part the Compiler.

The website's index pages (one for each node, plus one 'Master' page with links to the node indices) are re-generated from the data in the DBM::Deep DB hourly to show changes in utilization (queried on-the-fly from the RRD files) - I call this the Indexer

The actual target detail pages are displayed using 14all.cgi (Which reads the RRD files over the NFS mounts)


Anyhow... While my pollers are keeping up with polling the ~450 devices between them, the web server's Compiler is now choking on the sheer number of MRTG config files. In particular, the Compiler hits about 250MB of RAM usage and then stays there forever, never finishing the job.

It's getting stuck in the readcfg() routine from MRTG_Lib 
    print "Reading MRTG config from $main_mrtg_config...\n"; # DEBUG
    readcfg($main_mrtg_config,\@device,\%cfg,\%tgtcfg);

    print "Checking MRTG Config...\n"; # DEBUG
    cfgcheck(\@device, \%cfg, \%tgtcfg, \@target);

[download]
I mentioned earlier that I'm using DBM::Deep to store the final product... that was an add-on I made to my own code when it became clear that parsing the configs every hour was impossible, so I broke the Indexer in two, thus creating the Compiler.

I've tried tie-ing the %tgtcfg hash that readcfg() builds to another DBM::Deep DB but all it does is take an eternity at 100% CPU usage. (I killed it after 12 hours... and the file had grown to over 300MB)

I've tried using MLDBM with BerkeleyDB::Hash and Storable, but it just crashes with this error: 
ERROR: CFG Error in "setenv[192.168.52.1_1]", line 18:  must be XY="dd
+dd" AASD="
kjlkj" ...

[download]
(which is thrown by the code in MRTG_Lib)


I'm at wit's end (it wasn't a long way)... is there any magic I can work without having to completely re-write my Compiler?

Replies are listed 'Best First'.
Re: Parsing a really, REALLY big MRTG config
by educated_foo (Vicar) on Jun 07, 2007 at 18:56 UTC
    Are you talking about this? If so, I would try MRTG::Parse to see if it's faster. I would also look at reducing the horrors of "readcfg", which contains a lot of extraneous junk (not to mention foreach loops iterating over hash keys...). Favorite lines: 
            # oops spelling error
        s/^supress/suppress/gi;

    [download]
[reply]
[d/l]
      MRTG_lib... yep that's the one! I was typing from my frazzled and faulty memory :)

      I don't know how I missed that CPAN module! I guess since I already had Tobi's code I never bothered looking!

      I'm checking it out right now!
[reply]
Re: Parsing a really, REALLY big MRTG config
by shmem (Chancellor) on Jun 07, 2007 at 21:50 UTC
    Not knowing about specifics, but "parsing the configs every hour" strikes me as a very bad idea. Parse only that which has changed. Find the least memory / cpu stressing way to access information when it's needed. Think diff(1). 
    ERROR: CFG Error in "setenv[192.168.52.1_1]", line 18:  must be XY="dd
+dd" AASD="
kjlkj" ...

    [download]

    Far too little information to make an educated guess.

    --shmem 

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
[reply]
[d/l]
      Indeed... I hit a wall trying to parse all the configs each hour and thus created the script that loads, parses and stores the pre-digested data in a DBM::Deep file (Compiler), but only once a night. Then, the web-page generator (Indexer) could simply load into memory only what it needed, rather than the whole enchilada.

      DBM::Deep is great stuff, even for ridiculously deep/complex data structures... as long as you read the author's recommendations for performance (luckily I already coded it that way). It allowed me to make my code scale against the parsed MRTG configs *much* better by only adding a few lines and a tie! (but, alas, that is now over)
[reply]
Re: Parsing a really, REALLY big MRTG config
by NetWallah (Canon) on Jun 08, 2007 at 11:00 UTC
    For a long-term plan, you should consider upgrading the MRTG system to cacti.

    Cacti stores the MRTG config info in a MySQL database.

    Although the learning-curve for cacti is fairly steep, your MRTG background should help considerably. You will gain scalibility, efficiency, tons of features, and end-user friendliness.

         "An undefined problem has an infinite number of solutions." - Robert A. Humphrey         "If you're not part of the solution, you're part of the precipitate." - Henry J. Tillman

[reply]
      Thanks for the cacti suggestion... I tried rolling it out and it was immediately rejected by everyone who used it. Really, I think it was one of those odd, NIH-sort-of things, telco engineers being afraid of something they don't already know.

      Luckily for me... I just need to make this system work for one more quarter... and we'll be rolling out a brand-new polling solution company-wide that should banish MRTG forever. Hence my reluctance to re-write the current system! :)
[reply]

Back to Seekers of Perl Wisdom