Re: Perl script speed

In any optimisation task the first step is to profile the code. I guess that you have not yet done this (otherwise why would you not have shared the results with us).

Fortunately perl has a plethora of profiling options. The first port of call is the recommended Devel::NYTProf. Alternatively you could try the minimalist Devel::FastProf.

Once you've profiled it you will be in a better position to know where the problem lies.

In the meantime I would suggest breaking out your big nest of if/elses into separate subroutines for ease of both profiling and general maintenance.

Comment on Re: Perl script speed

Replies are listed 'Best First'.
Re^2: Perl script speed by rr27 (Initiate) on May 30, 2014 at 09:48 UTC
As evident, this is first time I am writing perl with SQLite, hence the inefficiency. I have done the profiling and got 1.DBI::st::execute as major time consumer. 2. DBD::SQLite::st::_prepare 3.DBI::_setup_handle. Please suggest ways to reduce the time.	[reply]
Re^3: Perl script speed by poj (Abbot) on May 30, 2014 at 11:50 UTC
Try switching AutoCommit off and then use `$dbh->commit()` after say every 1000 inserts, adjusting the number to get the best result. This node might help Loading bulk data into SQLite poj	[reply] [d/l]
Re^3: Perl script speed by Laurent_R (Canon) on May 30, 2014 at 10:42 UTC
I would have expected that parsing a 34 MB file would not take very much time (at least not anything near 30 minutes) and that the DB inserts would likely be the big time consumers (but it is of course much better to check it with profiling tools). Well, davido has suggested some possible improvements here.	[reply]