Parsing large File remotely

lazyreader has asked for the wisdom of the Perl Monks concerning the following question:

It is not an issue of connection with the server! I am looking for ways to perform string match over a large log file. I tried the following over a telnet object and my %parseLogData contains more than one string to be matched.

  my $cmd="cat $logName";

    @content = $self->{conn}->cmd($cmd);

    unless ( @content = $self->{conn}->cmd($cmd)){
        $logger->error(__PACKAGE__ . ".$sub_name: Failed to execute th
+e shell command:$cmd ");
        $logger->debug(__PACKAGE__ . ".$sub_name: <-- Leaving Sub [0]"
+);
        return 0;

        for my $i ( 0 .. $#{$parseLogData{$_}}) {
        my $parseString = $parseLogData{$_}[$i];

    my $flag = 0;

    if (@line = grep{/\Q$parseString\E/i} @content)
    {
        $logger->debug(__PACKAGE__ . ".$sub_name: PARSE SUCCESS: Expec
+ted -> \"$parseString\" in \"$logName\": Count of Matches -> $#line")
+;
        $flag = 1;
    }
    else {
        $logger->debug(__PACKAGE__ . ".$sub_name: PARSE FAILED: Expect
+ed :: -> \"$parseString\" in \"$logName\" ");
        }
    }
[download]

The above code works well for smaller log files but fails over large file sizes. other approach was to copy the file locally and perform parsing, which is not possible due to server restriction to establish FTP session. Hence I need to know any other alternative for my problem. Thanks.

Comment on Parsing large File remotely Download Code

Replies are listed 'Best First'.
Re: Parsing large File remotely by Corion (Patriarch) on Jan 28, 2010 at 17:43 UTC
If you don't have SSH access, you're out of luck. Personally, I would preparse the file on the server by launching Perl over SSH on the server to grep out the appropriate lines, and send those over the SSH connection to the client for further processing. Alternatively, consider a facility like `syslog`.	[reply] [d/l]
Re: Parsing large File remotely by JavaFan (Canon) on Jan 28, 2010 at 18:01 UTC
It seems your problem is access to the file; not so much the technical issue of how to parse a file. My advice is to talk to the people managing the box and see who they can give you access to the log file.	[reply]