in reply to Parsing a lot of data. Very slow. Need suggestions

Hmm, I haven't gone over this with a fine-tooth comb, but do you really need to have those `report pattern...` lines inside a for loop? The only variable inside those two lines is $date, which appears to be a function of other variables that are set outside the for loop. In other words, are you executing the exact same system call in those backticks each time through the loop? (Maybe the debugger or profiler would be able to tell you? I don't really know.) If that's the case, you might want to move those lines outside of the loop? Or maybe you have a reason for calling the same exact command several times. Or maybe I'm just missing the point completely. *shrug*

You also mentioned "SQL queries" and updating a database. There's no SQL in this script at all. Perhaps it's in one of the external programs? Regardless, if you're not using placeholders when updating your database, you may want to consider doing so. That'll save time when interacting with the database.

--

There are 10 kinds of people -- those that understand binary, and those that don't.

Replies are listed 'Best First'.
Re: Re: Parsing a lot of data. Very slow. Need suggestions
by RMGir (Prior) on Aug 15, 2002 at 23:10 UTC
    There's no SQL in this script at all.

    I wonder what 

    printf("update msnNewsletter set open='%d', click='%d' where dateCreat
+ed like '%d-%02d-%
+02d%%';\n", $opens, $transfers,$year,$month,$day);

    [download]
    is, then? :)
    --
    Mike
[reply]
[d/l]
Re: Re: Parsing a lot of data. Very slow. Need suggestions
by vxp (Pilgrim) on Aug 16, 2002 at 12:43 UTC
    its not the same command, because the $date variable gets changed each time the process is called. in this case here, its 120 dates. 120 different commands. and then 120 more, for another similar command. they need to be in a loop. :)
[reply]
      use place holders. There is a lot of overhead in your rdbms reparsing an sql statement...
[reply]

In Section Seekers of Perl Wisdom