in reply to Re^2: batch processing via single open
in thread batch processing via single open

Possible in a way?

OF course yes! That's why I asked about calling semantics. Which part is gathering the collection of files to iterate over? the shell script? do you want perl to gather the files? what system are you on?

<handwaving style="amount: lots"> If you are gathering the file via a shell script, you could do something along these lines 

( while read command param; do  # whatever method
    files=`command param`       # to gather the files
    for file in $files; do      # process files
        echo $file
    done
done) | maincode.pl

[download]

and in maincode.pl 

#!/usr/bin/perl
use IPC::Open2;
my $cmd = 'whatever'; # really, I have no idea what you are doing
$pid = open2(\*CHLD_OUT, \*CHLD_IN, $cmd) or die "oops: $!\n";

while(<STDIN>) {
    chomp;
    my $file = $_; # now do whatever with $file.
    open(I,'<', $file);
    while (my $line = <I>) {
        ... # do whatever with each line in the file
    }
}

[download]

but since I still don't know what you're up to, I can't give proper advice. See I know what I mean. Why don't you?

Maybe you really want some client/server stuff, or the perl code to act as a stream filter, and dispatch it's output to somewhere else. Each usage has different semantics; how can I know which are required, without a bit more of explanation from you?

--shmem 

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Replies are listed 'Best First'.
Re^4: batch processing via single open
by karden (Novice) on Jul 16, 2007 at 22:35 UTC
    No, no... We are very misdirected at the moment. We have nothing to do with reading/writing from/to files. Forget about it.

    I am not looking for a way to handle files. My shell script gathers the file names and passes to my Perl script. I do not want Perl to handle this instead, I am okay with it. Then my Perl file connects to "another program", executes some commands there (depending on the argument passed from shell to it) and exits. But this happens hundreds of thousands of times. Why not open the connection to "that program" only ONCE and use the same connection hundreds of thousands of times?

    I certainly understand you asking for detailed information and am quite aware of posting guidelines but this is a part of an experiment for a TREC proposal. So it is impossible for me to describe such a long thesis here. If I could not clarify what I am looking for so far, then that's the limitation of my English and we have nothing to do.

    One final try from me: Though I am not, let us assume the "external program" we are dealing is MySQL. That is: my Perl file opens an IPC connection to MySQL, selects a DB to work on and creates a table named ARGV[0]. That's the way it is, we cannot change it. Also assume that we want to create 100000 different tables. Now, why open a connection to MySQL for 100000 times? Let us open the connection and select the DB once at the beginning and then just create 100000 tables. It would save us a time of 99999*(time to open a new connection and select database), wouldn't it?

    Still does not make sense? :(

[reply]
      Still does not make sense?
      *shrug* Makes sense, for sure. With the MySQL abstraction at hand, go and apply that abstraction to your problem.

      It definitely doesn't make sense trying to help somebody who says "Please help me with doing something, but I won't tell you what it is".

      That said, another attempt: of course you can keep a connection to "some program" open, but only as long as the lifetime of your current perl process lasts. You can feed multiple file names to that program, but only if the calling semantics of that program support that way of feeding data, i.e. if it reads from STDIN. If your invoked program only handles arguments on its command line, there's no way, because you cannot extend the argument list after the fact.

      --shmem 

      _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
[reply]
        Ok, then I will have to open a new connection each time I run my code. That's all. I thank you so much for the trouble shmem!

        One correction though: I am somebody who says "Please help me with doing something; but while I really want to provide details I bet you, yourself, would not want to decode hundreds of lines of code". Thinking otherwise would be oversensitivity and mean insulting both you and me.

[reply]

In Section Seekers of Perl Wisdom