One possible suggestion would be to stop searching. Use an index of some kind. If your data can be flattened to "key maps to value" semantics, then a simple DBM will do. Otherwise, you should look at something like MySQL.

-- Randal L. Schwartz, Perl hacker

<assumptions>
 2 files x 65_000 lines x 60 fields x 8 chars = ~60MB data
 |-delimited ASCII
 search query is field 1
 data contains no 'escaped' |'s (e.g. \| or "xx|xx")
</assumptions>
[download]

Given the relatively 'small' (meaning under 128MB) database size, the questions would be 'how often do you need to search it?' and 'how often does the data change?'

As merlyn pointed out, you could index this file based upon whatever field(s) you are searching by; in this example, the ECL.

N.B. also that it would be simplest if (as one might assume) the data is only appended to (and never 'changed') to scan only the new data into the DB index. This example assumes that you get 'new files;' e.g. by rotating out files before indexing. If that is not the case (if the files are appended to in place), keeping track of the length of the file at the time it were indexed and then using seek() to begin indexing after the end would be more effective. (To 'reset' this index, just remove the db file.)

[mk_ecl_index]
#!/usr/bin/perl
use DB_File;
for my $filename (@ARGV)
{

 my %ecl;
 tie %ecl, DB_File, "$filename.db"
  or die "Can't tie $filename.db: $!";
 open ASCII, "<$filename";
 while (<INPUT>)
 {
  chomp;
  next unless m{^ ([^\|]*
                     \|
                   [^\|]*)
                     \|}x; # first two fields
  # can't store refs in basic DB_File
  # but data guaranteed not to contain \n, so... :-/
  $ecl{$1} = ''
   unless defined $ecl{$1};
  $ecl{$1} .= $_ . "\n";
 }
 close ASCII;
 untie %ecl;
}

[grep_ecl]
#!/usr/bin/perl
# n.b. args opposite of Unix grep; filename, query, q2...
my $filename = shift;
my %index;
tie %index, DB_File, "$filename.db"
 or die "Can't tie to $filename.db: $!";
for my $query (@ARGV)
{
  if ( exists $index{$query} )
  {
    print $index{$query}; # newlines already provided
  } else {
    print STDERR "$0: $filename: $query not found\n";
  }
}
untie %index;
[download]

File locking is left as an exercise for the reader; if you index in a cron job or logrotate script, you'll likely need it.

Hope that helps ;-)

use SuperSplit;

$AoA = supersplit_open('|','\n',$filename);

no strict;
for $line ( @AoA ){
  $ecl_hash->{$line[0]}->{$line[1]}->{$line[2]}=@line[3..$#line];
}
my $sub_ecl = $ecl_hash->{$NETID}->{$date};
some_function( $sub_ecl->{$time} ) for $time (keys %$sub_ecl);
[download]

open DATA, $filename;
my %ecl_hash;
my $str = "$NETID|$month/$date/$year";
while( <DATA> ){
  next if index($_,$str)<0;
  chomp;  
  my $item = (split '|' )[2];
  $ecl_hash{$item} = [] unless defined $ecl_hash{$item};
  push @$ecl_hash{$item}, $_;
}
[download]

Returns something similar, only the fields have not been separated into arrays. But maybe you don't even want to.

SuperSplit can be found here.

Cheers,

Jeroen
"We are not alone"(FZ)

Update: I ignored the fact that you want to search on the date/ID as well. Fixed in the last codeblock. Will be much faster, of course. What do you *really* want? You want to add an extra line to a report, with some sumary? You wanna make some graphs? Depending on the question, you can decide whether you want the whole thing read in, or just a little piece, or whether you'd better go and put everything in a database, as merlyn suggested...

Update2: after quite some CB, I think ImpalaSS'd better use the supersplit method and use the modified hash to access his data.

Hey,
Well to answer your question, what the program does is, for each site/per date/per half hour it grabs data from 2 different files. It then tkaes this data, and performs a lot of calculations to print numbers such as dropped calls, and total traffic etc. As of yet, no graphs or charts, all the data is just dumped into the arrays and then a subroutine takes the data and performs calculations and prints the results.

Dipul

As long as you don't want to do recalculations (as in deviations from the mean, or percentages from the max) you can stick with a database-less solution.

If you already know that you want the last lines of your database, why don't you just use ' open DATA, "tail -n $number $filename|";'? That would speed up things, as perl doesn't have to work the whole file.

Jeroen
"We are not alone"(FZ)

perl -ne ' foreach $item(@timearray) { $searchstring = "$NETID\|$month
+\/$date\/$year\|$item\|"; if (/"^$searchstring/) { print }}' /PHL/dat
+a1/PHL/tmp/ECL_STAT.txt
[download]

# Build a hash lookup for things I want, defaulting to not there
my %found;
my @search;
foreach my $item (@timearray) {
  my $id = "$NETID\|$month\/$date\/$year\|$item\|";
  push @search, $id;
  $found{$id} = "||$item||";
}
# Scan the file for them
my $file = "/PHL/data1/PHL/tmp/ECL_STAT";
open(ECL_STAT, "< $file") or die "Cannot read $file: $!";
while (<ECL_STAT>) {
  chomp;
  if (/(([^\|]+\|){3})/ and exists $found{$1}) {
    $found{$1} = $_;
  }
}
# Build output array
push @ECL, map $found{$_}, @search;
[download]

open( FILE, "/PHL/data1/PHL/tmp/ECL_STAT.txt" ) 
    or die "Couldn't open ECL_STAT.txt: $!";
my $found = 0;

foreach $item(@timearray){
      seek( FILE, 0, 0 );
      $searchstring = "$NETID\|$month\/$date\/$year\|$item\|";
      
      while( <FILE> ) {
          if( /^$searchstring/ ) {
            push @ECL, $_;
            $found = 1;
            last;
          }
      }

      push @ECL, "||$item||" unless $found;
}

close( FILE ) or die "Couldn't close ECL_STAT.txt: $!";
[download]

my %log_data;
open FH, $file or die "Can't open $file: $!";
while (<FH>) {
 my ($net_id, $mdy, $item) = split /\|/;
 $log_data{$net_id}{$mdy}{$item} = $_;
}

# Then cycle through data you're searching for
# and save/process it if its in the %log_data array
[download]

Update: Ok, I understand what you want, but you'd better put an eq in there instead of == as ANY string will match "" numerically (i.e. "a456" == "" is true). The only case where that won't be true is where $ECL contains a number or a string starting with a number which is then interpreted as a plain number in the numeric context, but all you are really trying to do is see if $ECL is empty, so use eq. If your data file is ever changed so the lines start with a non-number, your code will no longer work.

Actually if you ran your code with -w you'd get: Argument "" isn't numeric in numeric eq (==) at line xxx.

--
I'd like to be able to assign to an luser

Hey,
Basically with that line, its saying if a specific instance isnt found (for example is a half hour for a certain date, ie: 20042pa|02/06/2001|8:30 was missing) it would insert $item as a place carrier.

Dipul