Re: DBIx::Class. Cache table in memory

Two comments:

1) If sqlite isn't caching already, the operating system is. So you are not reading the authors from hard disk but from memory already

2) Is this an optimization that makes sense to look for? Even if you have one million records you want to store eventually, it will take half an hour combined. This half hour (presumably) is a one-time cost, not something you have to redo every day or every month.

It is a hard lesson to take (even though I preach it I don't follow this advice usually) but optimization should be done on a finished program and only after finding the hot spot (i.e. the lines of code where most time is spent during a typical program execution). And usually such a hot spot exists, so that 95% of the time is spent in only 5% of your code. It doesn't make much sense to optimize the other 95% of your code

Comment on Re: DBIx::Class. Cache table in memory

Replies are listed 'Best First'.
Re^2: DBIx::Class. Cache table in memory by zwon (Abbot) on Jan 13, 2010 at 21:06 UTC
If sqlite isn't caching already, the operating system is. In real application I'm using postgres, but yes, you're absolutely right, 10000 additional requests can't take 15 seconds, I had to think about it, thanks. I ran it with Devel::NYTProf and have found that it spends a lot of time inside Class::Accessor::Grouped and inside various DBIx::Class modules, but not in DBI and DBD::SQLite. So I guess this overhead is caused by creating resultsets for 'Artist'. Pity. Is this an optimization that makes sense to look for? Sure. I'm reworking log processing application, that has serious performance problems, and hoped to replace current DB abstraction layer with DBIx::Class, as our current custom module is not very flexible and would require a lot of work after I fixed a DB scheme. Concerning your last statement, generally I would agree with you, but in this case I know there the hot spots will be and I prefer to avoid them by spending some time on benchmarking and experimenting before I'll be limited by existing program structure. I think it will save me some time from these 95% in the end, so I could spend it here :)	[reply]

Replies are listed 'Best First'.

Re^2: DBIx::Class. Cache table in memory
by zwon (Abbot) on Jan 13, 2010 at 21:06 UTC

If sqlite isn't caching already, the operating system is.

In real application I'm using postgres, but yes, you're absolutely right, 10000 additional requests can't take 15 seconds, I had to think about it, thanks. I ran it with Devel::NYTProf and have found that it spends a lot of time inside Class::Accessor::Grouped and inside various DBIx::Class modules, but not in DBI and DBD::SQLite. So I guess this overhead is caused by creating resultsets for 'Artist'. Pity.

Is this an optimization that makes sense to look for?

Sure. I'm reworking log processing application, that has serious performance problems, and hoped to replace current DB abstraction layer with DBIx::Class, as our current custom module is not very flexible and would require a lot of work after I fixed a DB scheme.

Concerning your last statement, generally I would agree with you, but in this case I know there the hot spots will be and I prefer to avoid them by spending some time on benchmarking and experimenting before I'll be limited by existing program structure. I think it will save me some time from these 95% in the end, so I could spend it here :)

[reply]