Re^2: large perl module

Well, this is supposed to be 'lookup on demand' system providing lookup services to other servers.
Since it's doing it all the time, there is no point of loading the data over and over again from external storage.
The data was before in a data file (many of them actually), but that made the disk too busy.... It seems like these data files were not being cached on the OS level.
For each 10k lookups, perl had to read approx 2gb of files. 10k lookups were taking (when executed in a loop) 35s. Now, with the .pm loaded, it takes 10s.
The server has 8GB of ram, and it's using (according to top) only 3GB.

Comment on Re^2: large perl module

Replies are listed 'Best First'.
Re^3: large perl module by CountZero (Bishop) on Mar 05, 2010 at 07:11 UTC
providing lookup services to other servers A database server does exactly that and much more efficient than a "converted" web-server. Indexing, caching, ... are all highly optimised in a good database server. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]
Re^3: large perl module by jrsimmon (Hermit) on Mar 04, 2010 at 22:26 UTC
Moving the data to a data file doesn't mean loading it every time a request comes in. As pointed out below, having all the data in your .pm at best means you have it duplicated (loaded into memory in the .pm, and in whatever variable holds it). Worse, any changes to the data require changes to your source code, which is a design no-no.	[reply]