Re: Why can code be so slow?

Given that we have nothing about your code other than the CGI script names, I can't say for certain if this is a problem. But, in my experience, 90% of the bottlenecks in 90% of all Perl webapps backed by a database is the misuse of the database. If you are backed by a database, a few things to look at:

Are your queries using indices? An index can make the difference between 2 req/s and 200 req/s. (I have personally seen a 99% speed improvement several times.)
Are you using BLOBs to store files? If you are, get them out of the database and onto a NAS. Databases are horrible for storing binary files, such as pictures or audio, and should be used only as a last resort. Better to store the filenames in the database and the files on disk.
Are your tables properly normalized? I don't mean have you followed the rules of normalization to the 5th degree. I meant "Are your tables properly normalized for your needs?" I selectively denormalize for performance on a regular basis, but only after I've properly normalized everything.
Are you executing only those queries that are necessary? This is a Java example, but I once saw a 94% improvement in speed in specific screen in a Struts app by properly configuring Hibernate to not pull the sub-rows for the master query.

My criteria for good software:

Does it work?
Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Comment on Re: Why can code be so slow?

Replies are listed 'Best First'.
Re^2: Why can code be so slow? by freakingwildchild (Scribe) on May 02, 2007 at 15:43 UTC
I am using XML files because everything needs to be universally exportable to Flash and certain other applications where XML would be the easiest/universal solution. For meta information of a file I use inode.xml ; for settings for that current folder I use default.xml file etc.. I made the system that way I would never read more than 2 XML files maximally ; except when reading a full directory where I'd normally extract the metadata from the files I'd extract that info from XML files. That way I can also cross-export this info quick-and-easy to any other applications.. atleast.. I thougth so ;) My XML usage is sure not meant as database but rather as easy export/format for other systems working together with the data.	[reply]
Re^3: Why can code be so slow? by dragonchild (Archbishop) on May 02, 2007 at 15:58 UTC
In other words, you're hand-coding any searches you might need to do and iterating through datastructures that you create every time using an XML parser. And you're wondering why your system is slow? If you profile, I'll bet that the problem you're running into is in one or both of these places: Your XML parser is fast, but your usage isn't. 10-to-1 you're using the Tree option of XML::Parser, which is the slowest option. Once you have this data in memory, you're not working with them correctly. Algorithm choice is the second largest factor in runtime performance (after database (mis)usage). Also, if you're reading larger XML files (say, for your default.xml), then perl has to allocate RAM for the data structures. Depending on your OS, this could be up to 10% of your runtime. The proper solution, in case you're wondering, is to use some sort of database and export to XML as needed. So, use either something like MySQL (if you want a RDBMS and are comfortable with it) or a DBM solution could be DBM::Deep or BerkleyDB. In other words, solve the problem at hand first, then extend the solution. Baby steps. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]